Feaure matching API #5964

theNded · 2023-03-02T16:07:10Z

Expose correspondences from features api for global registration from external point-wise 3D features (e.g. FCGF).
Improve FPFH feature matching time:

Legacy	Tensor (CPU)	Tensor (CUDA)
0.023	0.016	0.0023

In theory a tensor RANSAC API could be included as well, but there are various problems. Problem analysis step by step.

Correctness: combination check all passed (L for legacy, T for tensor), no problem here:

FPFH feature	Feature matching	RANSAC	Status
L	L	L	✔️
L	T	L	✔️
T	L	L	✔️
T	T	L	✔️
T	T	T	✔️

Performance: tensor API is much slower.

Total registration time (s) (100K iterations)

Legacy	Tensor (CPU)	Tensor (CUDA)
0.088	36.04	35.74

Average run time breakdown (model selection/estimation/check happens every iteration; model validation only happens 250 times per 100K iterations)

	Legacy	Tensor (CPU)	Tensor (CUDA)
Model selection	10^-7	0.002	0.0020
Model estimation	2*10^-6	0.002	0.0022
Model check	10^-7	0.0012	0.0012
Model validation	0.003	0.0024	0.0045

Problem:

Model selection/estimation/check: Tensor point cloud operations (ComputeTransformation; SelectByIndex) are only suitable for large inputs. The tensor overhead is overwhelmingly large for small data.
Model validation: Hybrid NNS isn't faster for 3D on CUDA

Potential solutions:

This PR: Expose feature matching API first and use the legacy RANSAC solver. Tensor RANSAC is temporarily removed and archived in wei/ransac-tensor
Future work: Optimized (batch) operations for small point clouds, should consist of several PRs: batch SVD for transformation estimation; batch matrix multiplication for transformation; random tensor generation for model selection.

Type

Bug fix (non-breaking change which fixes an issue): Fixes #
New feature (non-breaking change which adds functionality). Resolves #
Breaking change (fix or feature that would cause existing functionality to not work as expected) Resolves #

Motivation and Context

Checklist:

I have run python util/check_style.py --apply to apply Open3D code style
to my code.
This PR changes Open3D behavior or adds new functionality.
- Both C++ (Doxygen) and Python (Sphinx / Google style) documentation is
  updated accordingly.
- I have added or updated C++ and / or Python unit tests OR included test
  results (e.g. screenshots or numbers) here.
I will follow up and update the code if CI fails.
For fork PRs, I have selected Allow edits from maintainers.

Description

This change is

…and debugging

update-docs · 2023-03-02T16:07:14Z

Thanks for submitting this pull request! The maintainers of this repository would appreciate if you could update the CHANGELOG.md based on your changes.

cpp/open3d/pipelines/registration/Feature.h

cpp/open3d/t/pipelines/registration/Feature.cpp

ShaoxiongYao · 2023-03-13T16:27:23Z

Thank you for providing this pull request! If I understand correctly, this will enable users to get correspondences from a set of features using the correspondences_from_features, right? With these features, I may be able to implement non-rigid ICP with my customized deformation solver.

theNded · 2023-03-13T17:22:31Z

Thank you for providing this pull request! If I understand correctly, this will enable users to get correspondences from a set of features using the correspondences_from_features, right? With these features, I may be able to implement non-rigid ICP with my customized deformation solver.

Yes, see
https://github.com/isl-org/Open3D/pull/5964/files#diff-6bdcf782f0d76f4eb437bd89ada11445e23a5008633f09131ab9edf8c5806998R118
CUDA acceleration is also supported with the tensor interface, I should update the example in a bit.

cpp/open3d/t/pipelines/registration/Feature.h

ssheorey · 2023-03-17T00:51:41Z

cpp/open3d/t/pipelines/registration/Feature.h

+/// of the aforementioned correspondence set where source[i] and target[j] are
+/// mutually the nearest neighbor. If the subset size is smaller than
+/// mutual_consistency_ratio * N, return the unfiltered set.
+core::Tensor CorrespondencesFromFeatures(const core::Tensor &source_features,


[Future PR is OK] Lowe ratio test as an alternative to mutual filter. This has a parameter to allow control over which matches are acceptable.

Sounds reasonable, will do in a future PR

ssheorey · 2023-03-17T00:58:45Z

examples/python/pipelines/registration_ransac.py

+
+        dst_fpfh_import = o3d.pipelines.registration.Feature()
+        dst_fpfh_import.data = dst_fpfh_np
+


To make this clearer, can we have a function that computes FPFH features and stores in an npy file. Then another function that reads from that file and uses the new API? A --save-features argument can control this behavior. This can be useful even just with FPFH if we are doing multiway registration [implementation not needed here].

I intend to extend this example with support to this functionality. However, there are inconsistencies between tensor point cloud downsample (nearest corner per voxel) and the legacy (mean per voxel). I will make another PR regarding t point cloud, and revisit this example.

cpp/open3d/pipelines/registration/Feature.cpp

ssheorey · 2023-03-17T01:06:36Z

cpp/open3d/t/pipelines/registration/Feature.cpp

@@ -90,6 +90,60 @@ core::Tensor ComputeFPFHFeature(const geometry::PointCloud &input,
    return fpfh;
 }

+core::Tensor CorrespondencesFromFeatures(const core::Tensor &source_features,


If these are CUDA tensors, should we copy them to the CPU first?

I think there is no need, as tensor nns backend is able to query properly (and fast). The conversion only has to be done for calling ransac.

cpp/open3d/t/pipelines/registration/Feature.cpp

guyuezuntinggithub · 2023-03-30T08:14:10Z

very great work! very appreciate for this.
can I use tensor FPFH in C++ API?

* expose correspondences from features api for external feature usages and debugging * Improve FPFH feature matching time

theNded added 2 commits March 2, 2023 11:01

expose correspondences from features api for external feature usages …

cf4c77c

…and debugging

apply style

15256c8

theNded force-pushed the wei/feature-matching branch from 54b1821 to dfab86f Compare March 2, 2023 17:28

add feature matching w/ tensor ablation

71489d9

theNded force-pushed the wei/feature-matching branch from dfab86f to 71489d9 Compare March 2, 2023 17:39

theNded closed this Mar 2, 2023

theNded reopened this Mar 3, 2023

rename and apply style

f04364b

theNded force-pushed the wei/feature-matching branch from 5d25e5e to f04364b Compare March 3, 2023 05:23

theNded added 2 commits March 6, 2023 21:08

add mutual filter and update docs

ea61fff

unit test and better default parameters

116c6e0

theNded marked this pull request as ready for review March 7, 2023 03:56

fix unit tests

a8e173b

theNded force-pushed the wei/feature-matching branch from 753114b to a8e173b Compare March 9, 2023 22:06

theNded requested review from yxlao and yuecideng March 10, 2023 00:35

yuecideng reviewed Mar 10, 2023

View reviewed changes

cpp/open3d/pipelines/registration/Feature.h Show resolved Hide resolved

yuecideng reviewed Mar 10, 2023

View reviewed changes

cpp/open3d/t/pipelines/registration/Feature.cpp Show resolved Hide resolved

theNded added 2 commits March 10, 2023 12:47

multi-thread for mutual_filter in tensor interface

5f28852

update doc

1c7b979

yuecideng approved these changes Mar 11, 2023

View reviewed changes

theNded requested a review from ssheorey March 13, 2023 15:18

ssheorey reviewed Mar 17, 2023

View reviewed changes

address review comments

2a7cd38

ssheorey approved these changes Mar 17, 2023

View reviewed changes

errissa merged commit a5be78c into master Apr 4, 2023

dbs4261 pushed a commit to dbs4261/Open3D that referenced this pull request Apr 13, 2023

Feaure matching API (isl-org#5964)

8a8a036

* expose correspondences from features api for external feature usages and debugging * Improve FPFH feature matching time

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feaure matching API #5964

Feaure matching API #5964

theNded commented Mar 2, 2023 •

edited

Loading

update-docs bot commented Mar 2, 2023

ShaoxiongYao commented Mar 13, 2023

theNded commented Mar 13, 2023

ssheorey Mar 17, 2023

theNded Mar 17, 2023

ssheorey Mar 17, 2023

theNded Mar 17, 2023

ssheorey Mar 17, 2023

theNded Mar 17, 2023

guyuezuntinggithub commented Mar 30, 2023


		dst_fpfh_import = o3d.pipelines.registration.Feature()
		dst_fpfh_import.data = dst_fpfh_np

Feaure matching API #5964

Feaure matching API #5964

Conversation

theNded commented Mar 2, 2023 • edited Loading

Type

Motivation and Context

Checklist:

Description

update-docs bot commented Mar 2, 2023

ShaoxiongYao commented Mar 13, 2023

theNded commented Mar 13, 2023

ssheorey Mar 17, 2023

Choose a reason for hiding this comment

theNded Mar 17, 2023

Choose a reason for hiding this comment

ssheorey Mar 17, 2023

Choose a reason for hiding this comment

theNded Mar 17, 2023

Choose a reason for hiding this comment

ssheorey Mar 17, 2023

Choose a reason for hiding this comment

theNded Mar 17, 2023

Choose a reason for hiding this comment

guyuezuntinggithub commented Mar 30, 2023

theNded commented Mar 2, 2023 •

edited

Loading