Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dynamic Voxelization on NuScenes #135

Closed
s-ryosky opened this issue Sep 28, 2020 · 7 comments
Closed

Dynamic Voxelization on NuScenes #135

s-ryosky opened this issue Sep 28, 2020 · 7 comments

Comments

@s-ryosky
Copy link

Describe the bug
I have changed the voxelization method in pointpillars/hv_pointpillars_fpn_sbn-all_4x8_2x_nus-3d.py from hard voxelization into dynamic voxelization.
Then an illegal memory access error was occurred.
Does the dynamic voxelization support nuscenes dataset?

Reproduction

  1. What command or script did you run?
    python ./tools/train.py ./configs/dynamic_voxelization/dv_pointpillars_fpn_sbn-all_4x8_2x_nus-3d.py

  2. Did you make any modifications on the code or config? Did you understand what you have modified?
    dv_pointpillars_fpn_sbn-all_4x8_2x_nus-3d.py was made based on pointpillars/hv_pointpillars_fpn_sbn-all_4x8_2x_nus-3d.py.
    The type of model was changed from MVXFasterRCNN to DynamicMVXFasterRCNN and the type of pts_voxel_encoder was changed from HardVFE to DynamicVFE.

  3. What dataset did you use?
    NuScenes dataset.

Environment
TorchVision: 0.7.0
OpenCV: 4.4.0
MMCV: 1.1.3
MMCV Compiler: GCC 7.3
MMCV CUDA Compiler: 10.1
MMDetection: 2.4.0
MMDetection3D: 0.6.0+30cec23

Error traceback
Traceback (most recent call last):
File "./tools/train.py", line 166, in
main()
File "./tools/train.py", line 162, in main
meta=meta)
File "/opt/conda/lib/python3.6/site-packages/mmdet/apis/train.py", line 143, in train_detector
runner.run(data_loaders, cfg.workflow, cfg.total_epochs)
File "/opt/conda/lib/python3.6/site-packages/mmcv/runner/epoch_based_runner.py", line 125, in run
epoch_runner(data_loaders[i], **kwargs)
File "/opt/conda/lib/python3.6/site-packages/mmcv/runner/epoch_based_runner.py", line 34, in train
**kwargs)
File "/opt/conda/lib/python3.6/site-packages/mmcv/parallel/data_parallel.py", line 67, in train_step
return self.module.train_step(*inputs[0], **kwargs[0])
File "/opt/conda/lib/python3.6/site-packages/mmdet/models/detectors/base.py", line 234, in train_step
losses = self(**data)
File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/shigenaka/workspace/mmdetection3d/mmdet3d/models/detectors/base.py", line 57, in forward
return self.forward_train(**kwargs)
File "/home/shigenaka/workspace/mmdetection3d/mmdet3d/models/detectors/mvx_two_stage.py", line 267, in forward_train
points, img=img, img_metas=img_metas)
File "/home/shigenaka/workspace/mmdetection3d/mmdet3d/models/detectors/mvx_two_stage.py", line 202, in extract_feat
pts_feats = self.extract_pts_feat(points, img_feats, img_metas)
File "/home/shigenaka/workspace/mmdetection3d/mmdet3d/models/detectors/mvx_faster_rcnn.py", line 52, in extract_pts_feat
voxels, coors, points, img_feats, img_metas)
File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/shigenaka/workspace/mmdetection3d/mmdet3d/models/voxel_encoders/voxel_encoder.py", line 267, in forward
voxel_feats, voxel_coors = self.vfe_scatter(point_feats, coors)
File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, *kwargs)
File "/home/shigenaka/workspace/mmdetection3d/mmdet3d/ops/voxel/scatter_points.py", line 113, in forward
points[inds], coors[inds][:, 1:])
File "/home/shigenaka/workspace/mmdetection3d/mmdet3d/ops/voxel/scatter_points.py", line 92, in forward_single
self.point_cloud_range)
File "/home/shigenaka/workspace/mmdetection3d/mmdet3d/ops/voxel/scatter_points.py", line 38, in forward
coors_range)
RuntimeError: CUDA error: an illegal memory access was encountered
terminate called after throwing an instance of 'c10::Error'
what(): CUDA error: an illegal memory access was encountered
Exception raised from create_event_internal at /opt/conda/conda-bld/pytorch_1595629416375/work/c10/cuda/CUDACachingAllocator.cpp:687 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x4d (0x7f073c9ca77d in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: c10::cuda::CUDACachingAllocator::raw_delete(void) + 0xb5d (0x7f073cc1ad9d in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::TensorImpl::release_resources() + 0x4d (0x7f073c9b6b1d in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #3: + 0x53e7fa (0x7f077624c7fa in /opt/conda/lib/python3.6/site-packages/torch/lib/libtorch_python.so)

frame #22: __libc_start_main + 0xf0 (0x7f079acc0830 in /lib/x86_64-linux-gnu/libc.so.6)

Aborted (core dumped)
@Tai-Wang
Copy link
Member

Have you ever tried to run the original config (hard voxelization version)? In addition, did you refer to the configs in the configs/dynamic_voxelization/? There are some changes in the voxel_layer and voxe_encoder setting. I would recommend you show your modified config.

@s-ryosky
Copy link
Author

The config is here.
I also changed the samples_per_gpu to 1 in nus-3d.py for debugging.

_base_ = [
    '../_base_/datasets/nus-3d.py', '../_base_/schedules/schedule_2x.py',
    '../_base_/default_runtime.py'
]

voxel_size = [5.0, 5.0, 6]
point_cloud_range = [-50, -50, -3, 50, 50, 3]
model = dict(
    type='DynamicMVXFasterRCNN',
    pts_voxel_layer=dict(
        max_num_points=-1,
        point_cloud_range=point_cloud_range,
        voxel_size=voxel_size,
        max_voxels=(-1, -1)),
    pts_voxel_encoder=dict(
        type='DynamicVFE',
        in_channels=4,
        feat_channels=[32, 64],
        with_distance=False,
        voxel_size=voxel_size,
        with_cluster_center=True,
        with_voxel_center=True,
        point_cloud_range=point_cloud_range,
        norm_cfg=dict(type='BN1d', eps=1e-3, momentum=0.01)),
    pts_middle_encoder=dict(
        type='PointPillarsScatter', in_channels=64, output_shape=[400, 400]),
    pts_backbone=dict(
        type='SECOND',
        in_channels=64,
        norm_cfg=dict(type='BN', eps=1e-3, momentum=0.01),
        layer_nums=[3, 5, 5],
        layer_strides=[2, 2, 2],
        out_channels=[64, 128, 128]),
    pts_neck=dict(
        type='FPN',
        norm_cfg=dict(type='BN', eps=1e-3, momentum=0.01),
        act_cfg=dict(type='ReLU'),
        in_channels=[64, 128, 128],
        out_channels=256,
        start_level=0,
        num_outs=3),
    pts_bbox_head=dict(
        type='Anchor3DHead',
        num_classes=10,
        in_channels=256,
        feat_channels=256,
        use_direction_classifier=True,
        anchor_generator=dict(
            type='AlignedAnchor3DRangeGenerator',
            ranges=[[-64, -64, -1.8, 64, 64, -1.8]],
            scales=[1, 2, 4],
            sizes=[
                [0.8660, 2.5981, 1.],  # 1.5/sqrt(3)
                [0.5774, 1.7321, 1.],  # 1/sqrt(3)
                [1., 1., 1.],
                [0.4, 0.4, 1],
            ],
            custom_values=[0, 0],
            rotations=[0, 1.57],
            reshape_out=True),
        assigner_per_size=False,
        diff_rad_by_sin=True,
        dir_offset=0.7854,  # pi/4
        dir_limit_offset=0,
        bbox_coder=dict(type='DeltaXYZWLHRBBoxCoder', code_size=9),
        loss_cls=dict(
            type='FocalLoss',
            use_sigmoid=True,
            gamma=2.0,
            alpha=0.25,
            loss_weight=1.0),
        loss_bbox=dict(type='SmoothL1Loss', beta=1.0 / 9.0, loss_weight=1.0),
        loss_dir=dict(
            type='CrossEntropyLoss', use_sigmoid=False, loss_weight=0.2)))
# model training and testing settings
train_cfg = dict(
    pts=dict(
        assigner=dict(
            type='MaxIoUAssigner',
            iou_calculator=dict(type='BboxOverlapsNearest3D'),
            pos_iou_thr=0.6,
            neg_iou_thr=0.3,
            min_pos_iou=0.3,
            ignore_iof_thr=-1),
        allowed_border=0,
        code_weight=[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.2, 0.2],
        pos_weight=-1,
        debug=False))
test_cfg = dict(
    pts=dict(
        use_rotate_nms=True,
        nms_across_levels=False,
        nms_pre=1000,
        nms_thr=0.2,
        score_thr=0.05,
        min_bbox_size=0,
        max_num=500))

@s-ryosky
Copy link
Author

I investigated, but the reason for the error may be out of memory.
And I found another issue is the dynamic voxelization requires a lot of memory capacity.

@Tai-Wang
Copy link
Member

I investigated, but the reason for the error may be out of memory.
And I found another issue is the dynamic voxelization requires a lot of memory capacity.

Yes, but there is also a minor bug in your config. Since you change the voxel size and point cloud range, you need to modify the output_shape in pts_middle_encode, which is very likely to lead to your CUDA error: an illegal memory access was encountered. In your case, it should be [20, 20].

BTW, I think the voxel size does not need to be so large if you just want to make it use less memory. For example, I guess [0.4, 0.4, 6] should be enough.

@s-ryosky
Copy link
Author

s-ryosky commented Sep 29, 2020 •
edited
Loading

I tried to change the config and then another error was occured.

_base_ = [
    '../_base_/datasets/nus-3d.py', '../_base_/schedules/schedule_2x.py',
    '../_base_/default_runtime.py'
]

voxel_size = [0.4, 0.4, 5]
point_cloud_range = [-51.2, -51.2, -3, 51.2, 51.2, 2]
model = dict(
    type='DynamicMVXFasterRCNN',
    pts_voxel_layer=dict(
        max_num_points=-1,
        point_cloud_range=point_cloud_range,
        voxel_size=voxel_size,
        max_voxels=(-1, -1)),
    pts_voxel_encoder=dict(
        type='DynamicVFE',
        in_channels=4,
        feat_channels=[32],
        with_distance=False,
        voxel_size=voxel_size,
        with_cluster_center=True,
        with_voxel_center=True,
        point_cloud_range=point_cloud_range,
        norm_cfg=dict(type='BN1d', eps=1e-3, momentum=0.01)),
    pts_middle_encoder=dict(
        type='PointPillarsScatter', in_channels=32, output_shape=[256, 256]),
    pts_backbone=dict(
        type='SECOND',
        in_channels=32,
        norm_cfg=dict(type='BN', eps=1e-3, momentum=0.01),
        layer_nums=[3, 5, 5],
        layer_strides=[2, 2, 2],
        out_channels=[64, 128, 128]),
    pts_neck=dict(
        type='FPN',
        norm_cfg=dict(type='BN', eps=1e-3, momentum=0.01),
        act_cfg=dict(type='ReLU'),
        in_channels=[64, 128, 128],
        out_channels=256,
        start_level=0,
        num_outs=3),
    pts_bbox_head=dict(
        type='Anchor3DHead',
        num_classes=10,
        in_channels=256,
        feat_channels=256,
        use_direction_classifier=True,
        anchor_generator=dict(
            type='AlignedAnchor3DRangeGenerator',
            ranges=[[-64, -64, -1.8, 64, 64, -1.8]],
            scales=[1, 2, 4],
            sizes=[
                [0.8660, 2.5981, 1.],  # 1.5/sqrt(3)
                [0.5774, 1.7321, 1.],  # 1/sqrt(3)
                [1., 1., 1.],
                [0.4, 0.4, 1],
            ],
            custom_values=[0, 0],
            rotations=[0, 1.57],
            reshape_out=True),
        assigner_per_size=False,
        diff_rad_by_sin=True,
        dir_offset=0.7854,  # pi/4
        dir_limit_offset=0,
        bbox_coder=dict(type='DeltaXYZWLHRBBoxCoder', code_size=9),
        loss_cls=dict(
            type='FocalLoss',
            use_sigmoid=True,
            gamma=2.0,
            alpha=0.25,
            loss_weight=1.0),
        loss_bbox=dict(type='SmoothL1Loss', beta=1.0 / 9.0, loss_weight=1.0),
        loss_dir=dict(
            type='CrossEntropyLoss', use_sigmoid=False, loss_weight=0.2)))
# model training and testing settings
train_cfg = dict(
    pts=dict(
        assigner=dict(
            type='MaxIoUAssigner',
            iou_calculator=dict(type='BboxOverlapsNearest3D'),
            pos_iou_thr=0.6,
            neg_iou_thr=0.3,
            min_pos_iou=0.3,
            ignore_iof_thr=-1),
        allowed_border=0,
        code_weight=[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.2, 0.2],
        pos_weight=-1,
        debug=False))
test_cfg = dict(
    pts=dict(
        use_rotate_nms=True,
        nms_across_levels=False,
        nms_pre=1000,
        nms_thr=0.2,
        score_thr=0.05,
        min_bbox_size=0,
        max_num=500))

2020-09-29 16:38:45,661 - mmdet - INFO - Start running, host: shigenaka@7ad97b789b1e, work_dir: /home/shigenaka/workspace/mmdetection3d/work_dirs/dv_pointpillars_fpn_sbn-all_4x8_2x_nus-3d
2020-09-29 16:38:45,661 - mmdet - INFO - workflow: [('train', 1)], max: 24 epochs
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [125,0,0], thread: [33,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [395,0,0], thread: [12,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [259,0,0], thread: [84,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [259,0,0], thread: [91,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [123,0,0], thread: [56,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [125,0,0], thread: [75,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [261,0,0], thread: [7,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [261,0,0], thread: [13,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [123,0,0], thread: [123,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [526,0,0], thread: [38,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [259,0,0], thread: [37,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [259,0,0], thread: [49,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [399,0,0], thread: [105,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [399,0,0], thread: [106,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [261,0,0], thread: [60,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [123,0,0], thread: [94,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [259,0,0], thread: [123,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [259,0,0], thread: [2,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [526,0,0], thread: [67,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [526,0,0], thread: [79,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [526,0,0], thread: [80,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [526,0,0], thread: [95,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [399,0,0], thread: [22,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [399,0,0], thread: [26,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [395,0,0], thread: [47,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [395,0,0], thread: [56,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [261,0,0], thread: [76,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [536,0,0], thread: [30,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [395,0,0], thread: [71,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [526,0,0], thread: [1,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [261,0,0], thread: [96,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [261,0,0], thread: [125,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [526,0,0], thread: [118,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [536,0,0], thread: [80,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [536,0,0], thread: [84,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [43,0,0], thread: [97,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [43,0,0], thread: [82,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [43,0,0], thread: [50,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [45,0,0], thread: [15,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [179,0,0], thread: [78,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [45,0,0], thread: [72,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [45,0,0], thread: [73,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [45,0,0], thread: [82,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [179,0,0], thread: [103,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [179,0,0], thread: [107,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [45,0,0], thread: [50,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [181,0,0], thread: [20,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [43,0,0], thread: [26,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [181,0,0], thread: [104,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [181,0,0], thread: [108,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [181,0,0], thread: [111,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [181,0,0], thread: [117,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [181,0,0], thread: [126,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [395,0,0], thread: [100,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [399,0,0], thread: [51,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [213,0,0], thread: [34,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [347,0,0], thread: [97,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [347,0,0], thread: [104,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [536,0,0], thread: [45,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [347,0,0], thread: [69,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [77,0,0], thread: [14,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [75,0,0], thread: [110,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [213,0,0], thread: [83,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [211,0,0], thread: [11,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [75,0,0], thread: [86,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [77,0,0], thread: [57,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [213,0,0], thread: [12,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [211,0,0], thread: [60,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [77,0,0], thread: [124,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [77,0,0], thread: [125,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [211,0,0], thread: [110,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [349,0,0], thread: [29,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [347,0,0], thread: [51,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [347,0,0], thread: [17,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [347,0,0], thread: [21,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [211,0,0], thread: [65,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [211,0,0], thread: [89,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [493,0,0], thread: [118,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [181,0,0], thread: [64,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [315,0,0], thread: [65,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [45,0,0], thread: [96,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [317,0,0], thread: [12,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [179,0,0], thread: [34,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [179,0,0], thread: [43,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [447,0,0], thread: [98,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [179,0,0], thread: [26,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [317,0,0], thread: [90,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [317,0,0], thread: [47,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [317,0,0], thread: [53,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [447,0,0], thread: [35,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [447,0,0], thread: [58,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [181,0,0], thread: [63,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [77,0,0], thread: [90,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [493,0,0], thread: [34,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [213,0,0], thread: [119,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [213,0,0], thread: [123,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [484,0,0], thread: [100,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [349,0,0], thread: [47,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [75,0,0], thread: [14,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [75,0,0], thread: [18,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [493,0,0], thread: [23,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [315,0,0], thread: [8,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [317,0,0], thread: [96,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [453,0,0], thread: [97,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [447,0,0], thread: [71,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [447,0,0], thread: [9,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [447,0,0], thread: [29,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [42,0,0], thread: [51,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [42,0,0], thread: [16,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [42,0,0], thread: [73,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [42,0,0], thread: [84,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [42,0,0], thread: [85,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [314,0,0], thread: [14,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [178,0,0], thread: [23,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [178,0,0], thread: [47,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [42,0,0], thread: [125,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [44,0,0], thread: [35,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [44,0,0], thread: [58,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [44,0,0], thread: [25,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [178,0,0], thread: [81,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [484,0,0], thread: [74,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [453,0,0], thread: [68,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [493,0,0], thread: [77,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [493,0,0], thread: [78,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [484,0,0], thread: [16,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [484,0,0], thread: [32,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [453,0,0], thread: [48,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [349,0,0], thread: [97,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [349,0,0], thread: [107,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [349,0,0], thread: [79,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [349,0,0], thread: [94,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
Traceback (most recent call last):
File "./tools/train.py", line 166, in
main()
File "./tools/train.py", line 162, in main
meta=meta)
File "/opt/conda/lib/python3.6/site-packages/mmdet/apis/train.py", line 143, in train_detector
runner.run(data_loaders, cfg.workflow, cfg.total_epochs)
File "/opt/conda/lib/python3.6/site-packages/mmcv/runner/epoch_based_runner.py", line 125, in run
epoch_runner(data_loaders[i], **kwargs)
File "/opt/conda/lib/python3.6/site-packages/mmcv/runner/epoch_based_runner.py", line 34, in train
**kwargs)
File "/opt/conda/lib/python3.6/site-packages/mmcv/parallel/data_parallel.py", line 67, in train_step
return self.module.train_step(*inputs[0], **kwargs[0])
File "/opt/conda/lib/python3.6/site-packages/mmdet/models/detectors/base.py", line 234, in train_step
losses = self(**data)
File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/shigenaka/workspace/mmdetection3d/mmdet3d/models/detectors/base.py", line 57, in forward
return self.forward_train(**kwargs)
File "/home/shigenaka/workspace/mmdetection3d/mmdet3d/models/detectors/mvx_two_stage.py", line 267, in forward_train
points, img=img, img_metas=img_metas)
File "/home/shigenaka/workspace/mmdetection3d/mmdet3d/models/detectors/mvx_two_stage.py", line 202, in extract_feat
pts_feats = self.extract_pts_feat(points, img_feats, img_metas)
File "/home/shigenaka/workspace/mmdetection3d/mmdet3d/models/detectors/mvx_faster_rcnn.py", line 52, in extract_pts_feat
voxels, coors, points, img_feats, img_metas)
File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/shigenaka/workspace/mmdetection3d/mmdet3d/models/voxel_encoders/voxel_encoder.py", line 262, in forward
point_feats = vfe(features)
File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/container.py", line 117, in forward
input = module(input)
File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, *kwargs)
File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/linear.py", line 91, in forward
return F.linear(input, self.weight, self.bias)
File "/opt/conda/lib/python3.6/site-packages/torch/nn/functional.py", line 1676, in linear
output = input.matmul(weight.t())
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling cublasCreate(handle)
terminate called after throwing an instance of 'c10::Error'
what(): CUDA error: device-side assert triggered
Exception raised from create_event_internal at /opt/conda/conda-bld/pytorch_1595629416375/work/c10/cuda/CUDACachingAllocator.cpp:687 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x4d (0x7f7fe997b77d in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: c10::cuda::CUDACachingAllocator::raw_delete(void) + 0xb5d (0x7f7fe9bcbd9d in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::TensorImpl::release_resources() + 0x4d (0x7f7fe9967b1d in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #3: + 0x53e7fa (0x7f80231fd7fa in /opt/conda/lib/python3.6/site-packages/torch/lib/libtorch_python.so)

frame #22: __libc_start_main + 0xf0 (0x7f8047c71830 in /lib/x86_64-linux-gnu/libc.so.6)

Aborted (core dumped)

@Tai-Wang
Copy link
Member

Tai-Wang commented Sep 29, 2020 •
edited
Loading

Please remember to change the range for anchors when the point cloud range is modified. In addition, the point cloud range in the dataset and model config should be both modified correspondingly.

@s-ryosky
Copy link
Author

Thank you for your help.
I modified the point cloud range and anchor range.

And the problem was solved after modifying the voxel size and sweeps_num to fit in memory capacity.
When sweeps_num is set to 10, DynamicScatter requires a lot of memory capacity because the number of points are large.

@manonthegithub manonthegithub mentioned this issue Mar 4, 2021
Closed
@Tai-Wang Tai-Wang mentioned this issue Jan 19, 2022
Closed
tpoisonooo pushed a commit to tpoisonooo/mmdetection3d that referenced this issue Sep 5, 2022
@SingleZombie
[Unittest]: Test mmcls (open-mmlab#135)
e240c15 
* Add tests

* lint

* add data

* Remove redundant code
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@s-ryosky @Tai-Wang