Forward-merge branch-25.02 into branch-25.04 #657
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![rapids-bot[bot]](https://avatars.githubusercontent.com/in/85914?s=60&v=4){kind=small}
Enables telemetry during CUVS CI runs. This is done by parsing GitHub Actions run log metadata and should have no impact on build or test times. xref rapidsai/build-infra#139 Authors: - Gil Forsyth (https://github.com/gforsyth) Approvers: - Mike Sarahan (https://github.com/msarahan) - Corey J. Nolet (https://github.com/cjnolet) URL: #652
|
FAILURE - Unable to forward-merge due to an error, manual merge is necessary. Do not use the IMPORTANT: When merging this PR, do not use the auto-merger (i.e. the |
This change reworks the api to allow it to be used with Java 21. The implementation is moved to an internal package, compiled with JDK 22, and packaged as an mrjar. The benefit of this structure is that the api can be used in environments that compile to a minimum of Java 21, but run on more recent JDKs like 22 and 23 - which is exactly what Elasticsearch and Lucene do. In fact, a minimum compilation target of Java 21 is common, since 21, at the time of writing, is the most recent LTS Java release. The most significant change is that the non-trivial api types are now, for the most part, interfaces. Instance can be created by one of the factory methods, which lookup an spi to find the implementation. If on a release greater than Java 21, then a functioning implementation is returned. Otherwise, a no-op implementation is returned. This is a reasonably standard way for a Java api to behave, and allows the developer to handle the case where the platform does not have a functioning implementation. This change also refactors the native downcall method handles so that they are static final constants - which optimise better by the JVM. It's also the generally accepted pattern, where the handles are tied to the lifetime of class which effectively mediates access - by virtue of reachability. Another thing that I added is the ability to programmatically set the temporary directory used for intermediate operations - this is important to how both Lucene and Elasticsearch work - since they commonly only have permission to write to certain parts of the disk. Additionally, 1. the error codes from native calls are plumbed in and checked. As well as `cuvsGetLastErrorText`. 2. a state is added to any classes that hold a reference to native resources that could be released. 3. a local arena is used for memory allocation only needed per downcall invocation, e.g. the return value. 4. I moved the tests to be integration tests, since they need to run on the jar (rather than the exploded classes). They can be run by any of; `mvn verify`, or `mvn integration-test`, or `mvn -Dit.test="*Hnsw*" verify` 5. I refactored the entry-points to the api to be static methods and added an `spi` layer. You can see the minimal impact on the tests. 6. Move the native library out of the top-level directory in the jar and into an os/arch position in the META-INF. 7. add service provider support for custom implementations. Authors: - Chris Hegarty (https://github.com/ChrisHegarty) Approvers: - Corey J. Nolet (https://github.com/cjnolet) URL: #628
Authors: - Ben Frederickson (https://github.com/benfred) Approvers: - Corey J. Nolet (https://github.com/cjnolet) URL: #660
Authors: - Divye Gala (https://github.com/divyegala) - Corey J. Nolet (https://github.com/cjnolet) Approvers: - Tamas Bela Feher (https://github.com/tfeher) - Corey J. Nolet (https://github.com/cjnolet) URL: #616
The ann-bench tool has been observed to deadlock on thread.join() due to unnecessary mutex lock. The problem is that the destructor doesn't release the thread mutex and thus doesn't allow the thread to escape the condition_variable.wait() function. The fix is to just remove the lock in the destructor (which doesn't modify the state of the task anyway). Authors: - Artem M. Chirkin (https://github.com/achirkin) Approvers: - Tamas Bela Feher (https://github.com/tfeher) URL: #667
This PR adds a Go API. It's far from completion and still work in progress. Feel free to suggest improvements! Authors: - Ajit Mistry (https://github.com/ajit283) - Corey J. Nolet (https://github.com/cjnolet) - Ben Frederickson (https://github.com/benfred) Approvers: - Ben Frederickson (https://github.com/benfred) - Gil Forsyth (https://github.com/gforsyth) URL: #212
Authors: - rhdong (https://github.com/rhdong) Approvers: - James Lamb (https://github.com/jameslamb) - Corey J. Nolet (https://github.com/cjnolet) - Ishan Chattopadhyaya (https://github.com/chatman) URL: #618
This PR adds docs for cuvs nn_descent Authors: - Severin Dicks (https://github.com/Intron7) Approvers: - Corey J. Nolet (https://github.com/cjnolet) URL: #668
… copy beyond 4B elems (#671) ann-bench keeps data dimensions as `uint32_t`. We use `std::fread` to copy the data from a file to the host memory and pass `n_rows * n_cols` there, which gets casted to size_t only after the multiplication. This leads to integer overflow for the datasets larger than 4B elements and a partial data copy. This PR fixes the bug by casting the dimensions before the multiplication. The bug only affects the benchmark cases where the data is requested in the host memory not backed by a file. Authors: - Artem M. Chirkin (https://github.com/achirkin) Approvers: - Tamas Bela Feher (https://github.com/tfeher) URL: #671
Authors: - Tarang Jain (https://github.com/tarang-jain) Approvers: - Corey J. Nolet (https://github.com/cjnolet) URL: #670
Authors: - Ben Frederickson (https://github.com/benfred) Approvers: - Corey J. Nolet (https://github.com/cjnolet) URL: #664
PR does the following:
- [x] Modifies CI to run pytest and e2e test of cuvs-bench
- [x] We need to test the additional time needed to run the tests. They should be fast, but if they are not, then we can add an additional job to run them in parallel.
- [x] Adds synthetic test-data generation so the CI jobs don't depend on downloading datasets, and users can have easy testing locally.
- [ ] Few improvements to be done to docs, yaml and other things to make it easy for users.
- [x] Check in some additional pytests that hadn't been checked in before.
Authors:
- Dante Gama Dessavre (https://github.com/dantegd)
- Corey J. Nolet (https://github.com/cjnolet)
- Micka (https://github.com/lowener)
Approvers:
- James Lamb (https://github.com/jameslamb)
- Corey J. Nolet (https://github.com/cjnolet)
URL: #574
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## branch-25.04 #657 +/- ##
===============================================
Coverage ? 84.00%
===============================================
Files ? 18
Lines ? 125
Branches ? 0
===============================================
Hits ? 105
Misses ? 20
Partials ? 0 ☔ View full report in Codecov by Sentry. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Forward-merge triggered by push to branch-25.02 that creates a PR to keep branch-25.04 up-to-date. If this PR is unable to be immediately merged due to conflicts, it will remain open for the team to manually merge. See forward-merger docs for more info.