Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a field type for high-dimensional bit vectors. #48322

Closed
jtibshirani opened this issue Oct 21, 2019 · 2 comments · Fixed by #110059
Closed

Add a field type for high-dimensional bit vectors. #48322

jtibshirani opened this issue Oct 21, 2019 · 2 comments · Fixed by #110059
Assignees
Labels
>enhancement :Search Relevance/Vectors Vector search Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch

Comments

@jtibshirani
Copy link
Contributor

jtibshirani commented Oct 21, 2019

The dense_vector type helps users work with vector 'embeddings' of unstructured data like text and images. This issue proposes to add a new 'bit vector' type and 'hamming distance' script function as part of supporting this use case.

Dense vector fields allow for storing float vectors. For images, it also seems common to use bit vectors:

There has also been recent work on converting traditional text embeddings to bit vectors, for example Learning Compressed Sentence Representations for On-Device Text Processing.

Compared to using a dense_vector to represent the binary vectors, a dedicated 'bit vector' type would require less space and could support faster distance computations. Looking forward, it may also be possible to support retrieval based on bit vector distance through a specialized strategy (distinct from what we've considered for float vectors in #42326).

@jtibshirani jtibshirani added >feature :Search/Search Search-related issues that do not fall into other categories labels Oct 21, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-search (:Search/Search)

@rjernst rjernst added the Team:Search Meta label for search team label May 4, 2020
@jtibshirani jtibshirani mentioned this issue Oct 4, 2021
17 tasks
@jtibshirani jtibshirani added :Search Relevance/Vectors Vector search and removed :Search/Search Search-related issues that do not fall into other categories labels Jul 21, 2022
@elastic elastic deleted a comment from elasticsearchmachine Jul 21, 2022
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-search (Team:Search)

@benwtrent benwtrent self-assigned this Jun 25, 2024
@javanna javanna added Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch and removed Team:Search Meta label for search team labels Jul 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>enhancement :Search Relevance/Vectors Vector search Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants