[Feature] Add spconv ops from mmdet3d #1581

wHao-Wu · 2021-12-13T03:50:03Z

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Motivation

Please describe the motivation of this PR and the goal you want to achieve through this PR.
Add spconv cuda operation from mmdet3d (branch: v1.0.0.dev0).

Modification

Please briefly describe what modification is made in this PR.
Add several files in mmcv/ops folder.

BC-breaking (Optional)

Does the modification introduce changes that break the backward-compatibility of the downstream repositories?
If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.

Use cases (Optional)

If this PR introduces a new feature, it is better to list some use cases here, and update the documentation.

Checklist

Before PR:

I have read and followed the workflow indicated in the CONTRIBUTING.md to create this PR.
Pre-commit or linting tools indicated in CONTRIBUTING.md are used to fix the potential lint issues.
Bug fixes are covered by unit tests, the case that causes the bug should be added in the unit tests.
New functionalities are covered by complete unit tests. If not, please add more unit test to ensure the correctness.
The documentation has been modified accordingly, including docstring or example tutorials.

After PR:

If the modification has potential influence on downstream or other related projects, this PR should be tested with some of those projects, like MMDet or MMCls.
CLA has been signed and all committers have signed the CLA in this PR.

CLAassistant · 2021-12-13T04:00:11Z

All committers have signed the CLA.

…ops-spconv

… add-mmdet3d-ops-spconv

zhouzaida · 2021-12-21T08:39:11Z

Please @AllentDan have a look.

mmcv/ops/csrc/parrots/modulated_deform_conv_pytorch.h

mmcv/ops/csrc/common/spconv/prettyprint.h

mmcv/ops/csrc/common/spconv/paramsgrid.h

mmcv/ops/csrc/common/spconv/spconv/geometry.h

mmcv/ops/csrc/common/spconv/spconv/indice.h

mmcv/ops/csrc/common/spconv/paramsgrid.h

mmcv/ops/csrc/common/spconv/tensorview/tensorview.h

mmcv/ops/sparse_modules.py

teamwong111

Some lint and unit test failures need fix.

mmcv/ops/csrc/common/spconv/spconv/indice.cu.h

mmcv/ops/csrc/pytorch/sparse_indice_cpu.cpp

mmcv/ops/csrc/pytorch/sparse_pool_ops.cpp

mmcv/ops/csrc/pytorch/cuda/fused_spconv_ops_cuda.cu

mmcv/ops/csrc/pytorch/pybind.cpp

mmcv/ops/csrc/pytorch/spconv_ops.cpp

mmcv/ops/csrc/common/spconv/tensorview/helper_launch.h

mmcv/ops/csrc/common/spconv/tensorview/helper_kernel.cu.h

(2) Switch `T` to `scalar_t` (3) Remove useless lines (4) Refine example in docstring of sparse_modules.py

(2) remove useless files (3) move cpu files to `pytorch/cpu`

grimoire · 2021-12-30T06:36:09Z

mmcv/ops/csrc/common/pytorch_cpp_helper.hpp

@@ -1,6 +1,6 @@
 #ifndef PYTORCH_CPP_HELPER
 #define PYTORCH_CPP_HELPER
-#include <torch/extension.h>
+#include <torch/types.h>


Do we have to modify this?

Yes. If not, the error error: invalid static_cast from type ‘const torch::OrderedDict<std::basic_string<char>, at::Tensor>’ to type ‘torch::OrderedDict<std::basic_string<char>, at::Tensor>&’ cannot be handled.

mmcv/ops/csrc/pytorch/cpu/pixel_group.cpp

mmcv/ops/sparse_functional.py

ZwwWayne · 2022-02-10T12:57:30Z

tests/test_cnn/test_build_layers.py

@@ -59,6 +59,9 @@ def test_build_conv_layer():
    assert layer.groups == kwargs['groups']
    assert layer.dilation == (kwargs['dilation'], kwargs['dilation'])

+    # some of sparse conv do not have arguments 'dilation' and 'group'
+    kwargs = dict(in_channels=4, out_channels=8, kernel_size=3)


Other conv layers should test this. Therefore, we cannot simply change the kwargs but only pop out the group number for spconv layers

The SparseInverseConv2d and SparseInverseConv3d here cannot take the dilation as input (although these classes inherit from the base class SparseConvolution, which can take the dilation as input). So there are some choices:

Pop out the dilation and group in kwargs;

Revise the interfaces of SparseInverseConv2d and `SparseInverseConv3d.

pop out dilation and group in kwarg only when the type is SparseInverseConv2d and SparseInverseConv3d. We should not pop it for all conv layers.

zhouzaida · 2022-02-18T02:28:28Z

Successfully compile and test with the following versions on windows

python36-pytorch1.5.0-cuda10.1
python37-pytorch1.8.0-cuda10.1
python38-pytorch1.10.0-cuda11.1

teamwong111 · 2022-02-18T03:07:15Z

Successfully compile and test with the following versions on Linux

python37-pytorch1.5.0-cuda9.0

wHao-Wu · 2022-02-18T03:12:51Z

Successfully compile and test with the following versions on Linux

python37-pytorch1.5.0-cuda10.1
python37-pytorch1.9.0-cuda10.2

grimoire · 2022-02-18T05:27:13Z

Successfully compile and test with the following versions on Linux

python37-pytorch1.8.0-cuda11.3

* Support deepcopy for Config (#1658) * Support deepcopy for Config * Iterate the `__dict__` of Config directly. * Use __new__ to avoid unnecessary initialization. * Improve according to comments * [Feature] Add spconv ops from mmdet3d (#1581) * add ops (spconv) of mmdet3d * fix typo * refactor code * resolve comments in #1452 * fix compile error * fix bugs * fix bug * transform from 'types.h' to 'extension.h' * fix bug * transform from 'types.h' to 'extension.h' in parrots * add extension.h in pybind.cpp * add unittest * Recover code * (1) Remove prettyprint.h (2) Switch `T` to `scalar_t` (3) Remove useless lines (4) Refine example in docstring of sparse_modules.py * (1) rename from `cu.h` to `cuh` (2) remove useless files (3) move cpu files to `pytorch/cpu` * reorganize files * Add docstring for sparse_functional.py * use dispatcher * remove template * use dispatch in cuda ops * resolve Segmentation fault * remove useless files * fix lint * fix lint * fix lint * fix unittest in test_build_layers.py * add tensorview into include_dirs when compiling * recover all deleted files * fix lint and comments * recover setup.py * replace tv::GPU as tv::TorchGPU & support device guard * fix lint Co-authored-by: hdc <hudingchang.vendor@sensetime.com> Co-authored-by: grimoire <yaoqian@sensetime.com> * Imporve the docstring of imfrombytes and fix a deprecation-warning (#1731) * [Refactor] Refactor the interface for RoIAlignRotated (#1662) * fix interface for RoIAlignRotated * Add a unit test for RoIAlignRotated * Make a unit test for RoIAlignRotated concise * fix interface for RoIAlignRotated * Refactor ext_module.nms_rotated * Lint cpp files * add transforms * add invoking time check for cacheable methods * fix lint * add unittest * fix bug in non-strict input mapping * fix ci * fix ci * fix compatibility with python<3.9 * fix typing compatibility * fix import * fix typing * add alternative for nullcontext * fix import * fix import * add docstrings * add docstrings * fix callable check * resolve comments * fix lint * enrich unittest cases * fix lint * fix unittest Co-authored-by: Ma Zerun <mzr1996@163.com> Co-authored-by: Wenhao Wu <79644370+wHao-Wu@users.noreply.github.com> Co-authored-by: hdc <hudingchang.vendor@sensetime.com> Co-authored-by: grimoire <yaoqian@sensetime.com> Co-authored-by: Jiazhen Wang <47851024+teamwong111@users.noreply.github.com> Co-authored-by: Hakjin Lee <nijkah@gmail.com>

* Support deepcopy for Config (open-mmlab#1658) * Support deepcopy for Config * Iterate the `__dict__` of Config directly. * Use __new__ to avoid unnecessary initialization. * Improve according to comments * [Feature] Add spconv ops from mmdet3d (open-mmlab#1581) * add ops (spconv) of mmdet3d * fix typo * refactor code * resolve comments in open-mmlab#1452 * fix compile error * fix bugs * fix bug * transform from 'types.h' to 'extension.h' * fix bug * transform from 'types.h' to 'extension.h' in parrots * add extension.h in pybind.cpp * add unittest * Recover code * (1) Remove prettyprint.h (2) Switch `T` to `scalar_t` (3) Remove useless lines (4) Refine example in docstring of sparse_modules.py * (1) rename from `cu.h` to `cuh` (2) remove useless files (3) move cpu files to `pytorch/cpu` * reorganize files * Add docstring for sparse_functional.py * use dispatcher * remove template * use dispatch in cuda ops * resolve Segmentation fault * remove useless files * fix lint * fix lint * fix lint * fix unittest in test_build_layers.py * add tensorview into include_dirs when compiling * recover all deleted files * fix lint and comments * recover setup.py * replace tv::GPU as tv::TorchGPU & support device guard * fix lint Co-authored-by: hdc <hudingchang.vendor@sensetime.com> Co-authored-by: grimoire <yaoqian@sensetime.com> * Imporve the docstring of imfrombytes and fix a deprecation-warning (open-mmlab#1731) * [Refactor] Refactor the interface for RoIAlignRotated (open-mmlab#1662) * fix interface for RoIAlignRotated * Add a unit test for RoIAlignRotated * Make a unit test for RoIAlignRotated concise * fix interface for RoIAlignRotated * Refactor ext_module.nms_rotated * Lint cpp files * add transforms * add invoking time check for cacheable methods * fix lint * add unittest * fix bug in non-strict input mapping * fix ci * fix ci * fix compatibility with python<3.9 * fix typing compatibility * fix import * fix typing * add alternative for nullcontext * fix import * fix import * add docstrings * add docstrings * fix callable check * resolve comments * fix lint * enrich unittest cases * fix lint * fix unittest Co-authored-by: Ma Zerun <mzr1996@163.com> Co-authored-by: Wenhao Wu <79644370+wHao-Wu@users.noreply.github.com> Co-authored-by: hdc <hudingchang.vendor@sensetime.com> Co-authored-by: grimoire <yaoqian@sensetime.com> Co-authored-by: Jiazhen Wang <47851024+teamwong111@users.noreply.github.com> Co-authored-by: Hakjin Lee <nijkah@gmail.com>

* Support deepcopy for Config (#1658) * Support deepcopy for Config * Iterate the `__dict__` of Config directly. * Use __new__ to avoid unnecessary initialization. * Improve according to comments * [Feature] Add spconv ops from mmdet3d (#1581) * add ops (spconv) of mmdet3d * fix typo * refactor code * resolve comments in #1452 * fix compile error * fix bugs * fix bug * transform from 'types.h' to 'extension.h' * fix bug * transform from 'types.h' to 'extension.h' in parrots * add extension.h in pybind.cpp * add unittest * Recover code * (1) Remove prettyprint.h (2) Switch `T` to `scalar_t` (3) Remove useless lines (4) Refine example in docstring of sparse_modules.py * (1) rename from `cu.h` to `cuh` (2) remove useless files (3) move cpu files to `pytorch/cpu` * reorganize files * Add docstring for sparse_functional.py * use dispatcher * remove template * use dispatch in cuda ops * resolve Segmentation fault * remove useless files * fix lint * fix lint * fix lint * fix unittest in test_build_layers.py * add tensorview into include_dirs when compiling * recover all deleted files * fix lint and comments * recover setup.py * replace tv::GPU as tv::TorchGPU & support device guard * fix lint Co-authored-by: hdc <hudingchang.vendor@sensetime.com> Co-authored-by: grimoire <yaoqian@sensetime.com> * Imporve the docstring of imfrombytes and fix a deprecation-warning (#1731) * [Refactor] Refactor the interface for RoIAlignRotated (#1662) * fix interface for RoIAlignRotated * Add a unit test for RoIAlignRotated * Make a unit test for RoIAlignRotated concise * fix interface for RoIAlignRotated * Refactor ext_module.nms_rotated * Lint cpp files * add transforms * add invoking time check for cacheable methods * fix lint * add unittest * fix bug in non-strict input mapping * fix ci * fix ci * fix compatibility with python<3.9 * fix typing compatibility * fix import * fix typing * add alternative for nullcontext * fix import * fix import * add docstrings * add docstrings * fix callable check * resolve comments * fix lint * enrich unittest cases * fix lint * fix unittest Co-authored-by: Ma Zerun <mzr1996@163.com> Co-authored-by: Wenhao Wu <79644370+wHao-Wu@users.noreply.github.com> Co-authored-by: hdc <hudingchang.vendor@sensetime.com> Co-authored-by: grimoire <yaoqian@sensetime.com> Co-authored-by: Jiazhen Wang <47851024+teamwong111@users.noreply.github.com> Co-authored-by: Hakjin Lee <nijkah@gmail.com>

* Support deepcopy for Config (open-mmlab#1658) * Support deepcopy for Config * Iterate the `__dict__` of Config directly. * Use __new__ to avoid unnecessary initialization. * Improve according to comments * [Feature] Add spconv ops from mmdet3d (open-mmlab#1581) * add ops (spconv) of mmdet3d * fix typo * refactor code * resolve comments in open-mmlab#1452 * fix compile error * fix bugs * fix bug * transform from 'types.h' to 'extension.h' * fix bug * transform from 'types.h' to 'extension.h' in parrots * add extension.h in pybind.cpp * add unittest * Recover code * (1) Remove prettyprint.h (2) Switch `T` to `scalar_t` (3) Remove useless lines (4) Refine example in docstring of sparse_modules.py * (1) rename from `cu.h` to `cuh` (2) remove useless files (3) move cpu files to `pytorch/cpu` * reorganize files * Add docstring for sparse_functional.py * use dispatcher * remove template * use dispatch in cuda ops * resolve Segmentation fault * remove useless files * fix lint * fix lint * fix lint * fix unittest in test_build_layers.py * add tensorview into include_dirs when compiling * recover all deleted files * fix lint and comments * recover setup.py * replace tv::GPU as tv::TorchGPU & support device guard * fix lint Co-authored-by: hdc <hudingchang.vendor@sensetime.com> Co-authored-by: grimoire <yaoqian@sensetime.com> * Imporve the docstring of imfrombytes and fix a deprecation-warning (open-mmlab#1731) * [Refactor] Refactor the interface for RoIAlignRotated (open-mmlab#1662) * fix interface for RoIAlignRotated * Add a unit test for RoIAlignRotated * Make a unit test for RoIAlignRotated concise * fix interface for RoIAlignRotated * Refactor ext_module.nms_rotated * Lint cpp files * add transforms * add invoking time check for cacheable methods * fix lint * add unittest * fix bug in non-strict input mapping * fix ci * fix ci * fix compatibility with python<3.9 * fix typing compatibility * fix import * fix typing * add alternative for nullcontext * fix import * fix import * add docstrings * add docstrings * fix callable check * resolve comments * fix lint * enrich unittest cases * fix lint * fix unittest Co-authored-by: Ma Zerun <mzr1996@163.com> Co-authored-by: Wenhao Wu <79644370+wHao-Wu@users.noreply.github.com> Co-authored-by: hdc <hudingchang.vendor@sensetime.com> Co-authored-by: grimoire <yaoqian@sensetime.com> Co-authored-by: Jiazhen Wang <47851024+teamwong111@users.noreply.github.com> Co-authored-by: Hakjin Lee <nijkah@gmail.com>

DCNSW and others added 4 commits November 2, 2021 23:05

add ops (spconv) of mmdet3d

5115f07

fix typo

a4d2197

refactor code

9c2c06a

resolve comments in open-mmlab#1452

55a0a15

WU-Wenhao and others added 11 commits December 13, 2021 13:15

resolve conflict

fa91f84

fix compile error

924058e

Merge branch 'master' of github.com:open-mmlab/mmcv into add-mmdet3d-…

d4c31fa

…ops-spconv

Merge branch 'add-mmdet3d-ops-spconv' of github.com:wHao-Wu/mmcv into…

701b052

… add-mmdet3d-ops-spconv

fix bugs

05aac3e

fix bug

1df25fc

transform from 'types.h' to 'extension.h'

b431e45

fix bug

04b2ae2

transform from 'types.h' to 'extension.h' in parrots

a26e89e

add extension.h in pybind.cpp

da74a1d

add unittest

e0ffb74

zhouzaida requested a review from grimoire December 21, 2021 08:37

zhouzaida reviewed Dec 21, 2021

View reviewed changes

mmcv/ops/csrc/parrots/modulated_deform_conv_pytorch.h Outdated Show resolved Hide resolved

zhouzaida requested a review from teamwong111 December 21, 2021 08:44

Recover code

d054696

AllentDan reviewed Dec 22, 2021

View reviewed changes

teamwong111 reviewed Dec 22, 2021

View reviewed changes

WU-Wenhao added 3 commits December 27, 2021 17:04

(1) Remove prettyprint.h

638be36

(2) Switch `T` to `scalar_t` (3) Remove useless lines (4) Refine example in docstring of sparse_modules.py

(1) rename from cu.h to cuh

28059cf

(2) remove useless files (3) move cpu files to `pytorch/cpu`

reorganize files

5efa302

grimoire reviewed Dec 30, 2021

View reviewed changes

zhouzaida reviewed Jan 2, 2022

View reviewed changes

mmcv/ops/csrc/pytorch/cpu/pixel_group.cpp Outdated Show resolved Hide resolved

zhouzaida reviewed Jan 2, 2022

View reviewed changes

mmcv/ops/sparse_functional.py Show resolved Hide resolved

zhouzaida reviewed Jan 2, 2022

View reviewed changes

mmcv/ops/sparse_functional.py Show resolved Hide resolved

grimoire and others added 9 commits January 12, 2022 12:34

remove template

53c4a7a

use dispatch in cuda ops

3cec16d

resolve Segmentation fault

b8e43a0

resolve conflict

0b8ae6b

remove useless files

60a7ea9

fix lint

49cc59e

fix lint

7406a5d

fix lint

4766b20

fix unittest in test_build_layers.py

ec416f9

filaPro mentioned this pull request Feb 9, 2022

add FCAF3D (SOTA in ScanNet, SUN RGB-D, S3DIS) open-mmlab/mmdetection3d#1219

Open

ZwwWayne reviewed Feb 10, 2022

View reviewed changes

WU-Wenhao added 4 commits February 15, 2022 17:34

add tensorview into include_dirs when compiling

4c3b0dd

recover all deleted files

bfb1676

fix lint and comments

d56f750

recover setup.py

11dc166

replace tv::GPU as tv::TorchGPU & support device guard

037f4d2

fix lint

03e4532

ZwwWayne merged commit c1de4c9 into open-mmlab:master Feb 18, 2022

Tai-Wang mentioned this pull request Jun 7, 2022

Progress of Migration of CUDA ops to MMCV open-mmlab/mmdetection3d#994

Closed

11 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] Add spconv ops from mmdet3d #1581

[Feature] Add spconv ops from mmdet3d #1581

wHao-Wu commented Dec 13, 2021

CLAassistant commented Dec 13, 2021 •

edited

Loading

zhouzaida commented Dec 21, 2021

teamwong111 left a comment

grimoire Dec 30, 2021

wHao-Wu Dec 30, 2021

ZwwWayne Feb 10, 2022

wHao-Wu Feb 15, 2022

ZwwWayne Feb 15, 2022

zhouzaida commented Feb 18, 2022 •

edited

Loading

teamwong111 commented Feb 18, 2022

wHao-Wu commented Feb 18, 2022 •

edited

Loading

grimoire commented Feb 18, 2022 •

edited

Loading

[Feature] Add spconv ops from mmdet3d #1581

[Feature] Add spconv ops from mmdet3d #1581

Conversation

wHao-Wu commented Dec 13, 2021

Motivation

Modification

BC-breaking (Optional)

Use cases (Optional)

Checklist

CLAassistant commented Dec 13, 2021 • edited Loading

zhouzaida commented Dec 21, 2021

teamwong111 left a comment

Choose a reason for hiding this comment

grimoire Dec 30, 2021

Choose a reason for hiding this comment

wHao-Wu Dec 30, 2021

Choose a reason for hiding this comment

ZwwWayne Feb 10, 2022

Choose a reason for hiding this comment

wHao-Wu Feb 15, 2022

Choose a reason for hiding this comment

ZwwWayne Feb 15, 2022

Choose a reason for hiding this comment

zhouzaida commented Feb 18, 2022 • edited Loading

teamwong111 commented Feb 18, 2022

wHao-Wu commented Feb 18, 2022 • edited Loading

grimoire commented Feb 18, 2022 • edited Loading

CLAassistant commented Dec 13, 2021 •

edited

Loading

zhouzaida commented Feb 18, 2022 •

edited

Loading

wHao-Wu commented Feb 18, 2022 •

edited

Loading

grimoire commented Feb 18, 2022 •

edited

Loading