Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Indexes on compressed data #3

Open
5 of 8 tasks
unroll opened this issue Nov 27, 2024 · 0 comments
Open
5 of 8 tasks

Indexes on compressed data #3

unroll opened this issue Nov 27, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@unroll
Copy link
Owner

unroll commented Nov 27, 2024

Allow compressed indexes: an index that is built using compressed data, or built using the original that but stores compressed data for querying. Examples: Vanama+PQ, IVFPQ.

  • Adapter to tie an index and compressor
  • Internal changes to indexes and compressors to support compressed data storage
  • Build index from compressed data
  • Build index from uncompressed data, query on compressed data
  • Assymetric distance computation (ADC) for PQ and similar

Stretch goals (not required to close issue):

  • Optimize distance computation on compressed, uncompressed, and assymetric comparisons
  • Support compressors with preprocessing steps such as OPQ and similar
  • Improve BaseIndex / BaseCompressor APIs to support compressed indexes
@unroll unroll added the enhancement New feature or request label Nov 27, 2024
@unroll unroll changed the title Support indexes on compressed data (example: Vanama+PQ, IVFPQ, etc) Indexes on compressed data (example: Vanama+PQ, IVFPQ, etc) Nov 27, 2024
@unroll unroll changed the title Indexes on compressed data (example: Vanama+PQ, IVFPQ, etc) Indexes on compressed data Nov 27, 2024
unroll added a commit that referenced this issue Nov 27, 2024
Also provides get_compressed_d()
unroll added a commit that referenced this issue Nov 27, 2024
Includes assymetric distances, building from compressed or uncompressed data
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

When branches are created from issues, their pull requests are automatically linked.

1 participant