Add MST optimization to guarantee the connectivity of CAGRA graphs #237
Conversation
tfeher
added
feature request
There was a problem hiding this comment.
Thanks Akira for porting the PR to cuVS! The original PR rapidsai/raft#2218 was already reviewed, and I see that most of the issues are addressed here. I have a marked a few smaller issues below.
There is some scope for improving code readability by relying more on mdspan, and adding a helper function to check duplicates (e.g. here). @anaruse I could easily apply the mdspan changes on the PR, if you are fine with that.
|
Thanks for the review, Tamas. I have fixed the points you pointed out, and it would be helpful if you could make the mdspan related changes. |
|
Before this is merged can we please have a GitHub issue created to expose this through the public APIs? We already have a use-case for this API in HDBSCAN and single linkage clustering. |
|
Also, given that both of you have made changes to the PR, I’d like to have a 3rd person do a quick look over. I’ll take a look tomorrow. |
|
Just noticed this issue #256 was created, so disregard my first comment. |
|
Thanks Corey, more reviews are welcome! Note that I have only added a merge commit to this PR to bring it up to date with base branch. I planned to add mdspan usage related refactoring, but I decided to postpone that in a follow up (added a note in #256). This PR is already useful in its current form: it helps to exclude/debug one issue that can lead to low recall. That is why I would prefer to get it merged even with some pending refactoring. |
|
/ok to test |
|
/ok to test |
|
/ok to test |
|
/merge |
…apidsai#237) This PR allows us to guarantee the connectivity of the CAGRA search graph using approximate MST. It has been empirically shown that the graph indexes generated by CAGRA for search provide comparable search accuracy to other libraries, but reachability from any node to all nodes is not guaranteed. In fact, it has been confirmed that the number of strongly connected components (SCC) of graph indexes created by CAGRA is not 1 in some 100M scale datasets. This problem can be alleviated by increasing the number of degrees in the search graph, but this would increase the size of the graph index. It is desirable to address this problem without increasing the number of degrees of the search graph. Prior study has shown that this can be solved by using a Minimum Spanning Tree (MST)-like approach, but in general, MST calculation takes a long time. However, what is needed here is not an exact MST, but, for example, an approximate MST in which the total number of edges is not necessarily minimum. Such an approximate MST could be computed quickly on GPUs. This PR contains implementation to create a approximate MST on the GPU at high speed based on the above policy and use it to guarantee the connectivity of the search graph. This functionality is not always required, so it is considered an opt-in feature. A member variable named `guarantee_connectivity` is added to `index_params`, so set this variable to `true` if you wish to use this featgure. > cuvs::neighbors::cagra::index_params index_params; > index_params.guarantee_connectivity = true; > auto index = cuvs::neighbors::cagra::build(res, index_params, dataset_view); Authors: - Akira Naruse (https://github.com/anaruse) - Tamas Bela Feher (https://github.com/tfeher) - Corey J. Nolet (https://github.com/cjnolet) Approvers: - Tamas Bela Feher (https://github.com/tfeher) - Corey J. Nolet (https://github.com/cjnolet) URL: rapidsai#237
This PR allows us to guarantee the connectivity of the CAGRA search graph using approximate MST.
It has been empirically shown that the graph indexes generated by CAGRA for search provide comparable search accuracy to other libraries, but reachability from any node to all nodes is not guaranteed. In fact, it has been confirmed that the number of strongly connected components (SCC) of graph indexes created by CAGRA is not 1 in some 100M scale datasets.
This problem can be alleviated by increasing the number of degrees in the search graph, but this would increase the size of the graph index. It is desirable to address this problem without increasing the number of degrees of the search graph.
Prior study has shown that this can be solved by using a Minimum Spanning Tree (MST)-like approach, but in general, MST calculation takes a long time. However, what is needed here is not an exact MST, but, for example, an approximate MST in which the total number of edges is not necessarily minimum. Such an approximate MST could be computed quickly on GPUs.
This PR contains implementation to create a approximate MST on the GPU at high speed based on the above policy and use it to guarantee the connectivity of the search graph.
This functionality is not always required, so it is considered an opt-in feature. A member variable named
guarantee_connectivityis added toindex_params, so set this variable totrueif you wish to use this featgure.