Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DocDB] Enable SimSIMD in Usearch #23998

Open
1 task done
mbautin opened this issue Sep 18, 2024 · 0 comments
Open
1 task done

[DocDB] Enable SimSIMD in Usearch #23998

mbautin opened this issue Sep 18, 2024 · 0 comments
Assignees
@mbautin
Labels
area/docdb YugabyteDB core features kind/enhancement This is an enhancement of an existing feature priority/medium Medium priority issue

Comments

@mbautin
Copy link
Contributor

mbautin commented Sep 18, 2024 •
edited by jira bot
Loading

Jira Link: DB-12885

Description

Enable SimSIMD distance calculation in Usearch.

Issue Type

kind/enhancement

Warning: Please confirm that this issue does not contain any sensitive information

  • I confirm this issue does not contain any sensitive information.
@mbautin mbautin added area/docdb YugabyteDB core features status/awaiting-triage Issue awaiting triage labels Sep 18, 2024
@yugabyte-ci yugabyte-ci added kind/enhancement This is an enhancement of an existing feature priority/medium Medium priority issue labels Sep 18, 2024
mbautin added a commit that referenced this issue Sep 21, 2024
@mbautin
[#23998] DocDB: Update usearch and other header-only third-party depe…
e131b20 
…ndencies

Summary:
Updating inline third-party dependencies:

- usearch : https://github.com/unum-cloud/usearch/commits/240fe9c298100f9e37a2d7377b1595be6ba1f412
- fp16    : https://github.com/Maratyszcza/FP16/commits/98b0a46bce017382a6351a19577ec43a715b6835
- hnswlib : https://github.com/nmslib/hnswlib/commits/2142dc6f4dd08e64ab727a7bbd93be7f732e80b0
- simsimd : https://github.com/ashvardanian/simsimd/commits/6834f4639af0271e760b00c08a464555b536f4f4

Moving the auto-generated comments in files belonging to the inline thirdparty dependencies to the end of those files to preserve the original line numbers.

Not turning on SimSIMD use in Usearch yet because that has issues with GCC 11.

The updates to the third-party libraries will be auto-generated using the command below and pushed as separate commits:

build-support/thirdparty_tool --sync-inline-thirdparty
Jira: DB-12885

Test Plan:
Jenkins

Manual tests using hnsw_tool

Reviewers: tnayak, sergei

Reviewed By: tnayak

Subscribers: ybase

Differential Revision: https://phorge.dev.yugabyte.com/D38134
foucher pushed a commit that referenced this issue Sep 24, 2024
@yugabyte-ci @foucher
[BACKPORT pg15-cherrypicks] all: Bulk port from master - 107
218324e 
Summary:
 5d3e83e [PLAT-15199] Change TP API URLs according to latest refactoring
 a50a730 [doc][yba] YBDB compatibility (#23984)
 0c84dbe [#24029] Update the callhome diagnostics  not to send gflags details.
 b53ed3a [PLAT-15379][Fix PLAT-12510] Option to use UTC when dealing with cron exp. in backup schedule
 f0eab8f [PLAT-15278]: Fix DB Scoped XCluster replication restart
 344bc76 Revert "[PLAT-15379][Fix PLAT-12510] Option to use UTC when dealing with cron exp. in backup schedule"
 3628ba7 [PLAT-14459] Swagger fix
 bb93ebe [#24021] YSQL: Add --TEST_check_catalog_version_overflow
 9ab7806 [#23927] docdb: Add gflag for minimum thread stack size
 Excluded: 8c8adc0 [#18822] YSQL: Gate update optimizations behind preview flag
 5e86515 [#23768] YSQL: Fix table rewrite DDL before slot creation
 123d496 [PLAT-14682] Universe task should only unlock itself and make unlock aware of the lock config
 de9d4ad [doc][yba] CIS hardened OS support (#23789)
 e131b20 [#23998] DocDB: Update usearch and other header-only third-party dependencies
 1665662 Automatic commit by thirdparty_tool: update usearch to commit 240fe9c298100f9e37a2d7377b1595be6ba1f412.
 3adbdae Automatic commit by thirdparty_tool: update fp16 to commit 98b0a46bce017382a6351a19577ec43a715b6835.
 9a819f7 Automatic commit by thirdparty_tool: update hnswlib to commit 2142dc6f4dd08e64ab727a7bbd93be7f732e80b0.
 2dc58f4 Automatic commit by thirdparty_tool: update simsimd to tag v5.1.0.
 9a03432 [doc][ybm] Azure private link host (#24086)
 039c9a2 [#17378] YSQL: Testing for histogram_bounds in pg_stats
 09f7a0f [#24085] DocDB: Refactor HNSW wrappers
 555af7d [#24000] DocDB: Shutting down shared exchange could cause TServer to hang
 5743a03 [PLAT-15317]Alert emails are not in the correct format.
 8642555 [PLAT-15379][Fix PLAT-12510] Option to use UTC when dealing with cron exp. in backup schedule
 253ab07 [PLAT-15400][PLAT-15401][PLAT-13051] - Connection pooling ui issues and other ui issues
 57576ae [#16487] YSQL: Fix flakey TestPostgresPid test
 bc8ae45 Update ports for CIS hardened (#24098)
 6fa33e6 [#18152, #18729] Docdb: Fix test TestPgIndexSelectiveUpdate
 cc6d2d1 [docs] added and updated cves (#24046)
 Excluded: ed153dc [#24055] YSQL: fix pg_hint_plan regression with executing prepared statement

Test Plan: Jenkins: rebase: pg15-cherrypicks

Reviewers: jason, jenkins-bot

Differential Revision: https://phorge.dev.yugabyte.com/D38322
mbautin added a commit that referenced this issue Oct 1, 2024
@mbautin
[#23998] Update third-party dependencies and enable SimSIMD in Usearch
6baf188 
Summary:
- Upgrade SimSIMD and Usearch
- Enable SimSIMD in Usearch in as many configurations as possible
- Add options to save index to a file and load it from a file in hnsw_tool

Test Plan:
Jenkins

Inline third-party dependencies would be added on to the diff using `build-support/thirdparty_tool --sync-inline-thirdparty` and tested on Jenkins before this diff is landed.

Reviewers: sergei

Reviewed By: sergei

Subscribers: ybase

Differential Revision: https://phorge.dev.yugabyte.com/D38349
timothy-e pushed a commit that referenced this issue Oct 2, 2024
@yugabyte-ci @timothy-e
[BACKPORT pg15-cherrypicks] all: Bulk port from master - 115
80673da 
Summary:
 79a00fd [PLAT-15307]fix sensitive info leaks via Gflags
 cd26c93 [DOC-487] Voyager 1.8.2 changes (#24177)
 fa91de7 [docs] Apache Hudi integration with YSQL  (#23888)
 586d337 Updating DynamoDB comparison (#24216)
 aad5695 [#18822] YSQL: Promote autoflag to skip redundant update operations
 fa38152 Fix UBI image: Add -y option to install of hostname
 6baf188 [#23998] Update third-party dependencies and enable SimSIMD in Usearch
 d57db29 Automatic commit by thirdparty_tool: update usearch to commit 191d9bb46fe5e2a44d1505ce7563ed51c7e55868.
 aab1a8b Automatic commit by thirdparty_tool: update simsimd to tag v5.4.3-yb-1.
 161c0c8 [PLAT-15279] Adding unix timestamp to the core dump
 17c45ff [#24217] YSQL: fill definition of a shell type requires catalog version increment
 037fac0 [DB-13062] yugabyted: added banner and get started component
 2eedabd [doc] Read replica connection load balancing support in JDBC Smart driver (#24006)
 62a6a32 [#21467, #24153] Docdb: Add Read sequences as of time - sequences support for clone part 2
 12de78e [PLAT-14954] added support for systemd-timesyncd
 4a07eb8 [#23988] YSQL: Skip a table for schema comparison if its schema does not change
 d3fd39f [doc][ybm] Add reasoning behind no access to yugabyte user #21105 (#23930)
 556ba8a [PLAT-15074] Install node agents on nodes for the marked universes for on-prem providers
 9beb6dc [#22710][#22707] yugabyted: Update voyager migrations list landing page. (#22834)
 6128137 [PLAT-15545] Simplify the frozen universe message for end user in YBA
 4e36b78 JDBC Driver version update to 42.3.5-yb-8 (#24241)
 254c979 [PLAT-15519]: Update xCluster sync to remove tables from YBA DB

Test Plan: Jenkins: rebase: pg15-cherrypicks

Reviewers: tfoucher, fizaa, telgersma

Differential Revision: https://phorge.dev.yugabyte.com/D38624
@yugabyte-ci yugabyte-ci removed the status/awaiting-triage Issue awaiting triage label Nov 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mbautin mbautin

Labels
area/docdb YugabyteDB core features kind/enhancement This is an enhancement of an existing feature priority/medium Medium priority issue
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mbautin @yugabyte-ci