Terms query can accept encoded terms input as bitmap #8133

bowenlan-amzn · 2024-08-30T09:35:38Z

Description

Document the newly added bitmap filtering feature in 2.17.

Issues Resolved

Closes #8121

Version

2.17

Checklist

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com>

github-actions · 2024-08-30T09:35:51Z

Thank you for submitting your PR. The PR states are In progress (or Draft) -> Tech review -> Doc review -> Editorial review -> Merged.

Before you submit your PR for doc review, make sure the content is technically accurate. If you need help finding a tech reviewer, tag a maintainer.

When you're ready for doc review, tag the assignee of this PR. The doc reviewer may push edits to the PR directly or leave comments and editorial suggestions for you to address (let us know in a comment if you have a preference). The doc reviewer will arrange for an editorial review.

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

kolchfa-aws · 2024-08-30T19:22:30Z

@bowenlan-amzn I have reviewed and pushed my edits into your PR. Please verify that everything is still technically correct (especially the expanded Python example: please make sure I'm using the correct library). Also, for the value_type, please provide the valid values and default value. Thank you!

bowenlan-amzn · 2024-09-04T03:22:51Z

_query-dsl/term/terms.md

@@ -39,6 +39,7 @@ Parameter | Data type | Description
 :--- | :--- | :---
 `<field>` | String | The field in which to search. A document is returned in the results only if its field value exactly matches at least one term, with the correct spacing and capitalization.
 `boost` | Floating-point | A floating-point value that specifies the weight of this field toward the relevance score. Values above 1.0 increase the field’s relevance. Values between 0.0 and 1.0 decrease the field’s relevance. Default is 1.0.
+`value_type` | String | Specifies the type of the values that are being used for filtering. Valid values are <>. Default is <>.


Valid values are default and bitmap. Default is default or just omit this parameter.

_query-dsl/term/terms.md

Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>

kolchfa-aws

LGTM. Thank you, @bowenlan-amzn! Applying "Done but waiting to merge" label to put this in the 2.17 editorial queue starting 9/11. This PR will be merged before the release.

natebower

@bowenlan-amzn @kolchfa-aws Please see my comments and changes and let me know if you have any questions. Thanks!

_query-dsl/term/terms.md

natebower · 2024-09-12T10:38:55Z

_query-dsl/term/terms.md

+The `terms` query can filter for multiple terms simultaneously. However, when the number of terms in the input filter increases to a large value (around 10,000), the resulting network and memory overhead can become significant, making the query inefficient. In such cases, consider encoding your large terms filter using a [roaring bitmap](https://github.com/RoaringBitmap/RoaringBitmap) for more efficient filtering. 
+
+For example, you have two indexes: the `products` index, which contains all the products a company sells, and the `customers` index, which stores filters representing customers who own specific products. 
+


Rather than "For example, you have two indexes:", I would prefer that this be something like "The following example [noun] assumes that you have two indexes:".

Applied. I don't think there's a noun needed because this is just an example (consisting of multiple requests).

natebower · 2024-09-12T10:42:53Z

_query-dsl/term/terms.md

+```
+{% include copy-curl.html %}
+
+For each customer, you need to generate a bitmap that represents the product IDs of the products the customer owns. This bitmap effectively encodes the filter criteria for that customer. To illustrate the process, you'll create a `terms` filter for a customer whose ID is `customer123` and who owns products `111`, `222`, and `333`.


Second sentence: The user won't be illustrating the process to anyone by doing this. "In this example [noun], you'll..."?

_query-dsl/term/terms.md

natebower · 2024-09-12T10:44:24Z

_query-dsl/term/terms.md

+For each customer, you need to generate a bitmap that represents the product IDs of the products the customer owns. This bitmap effectively encodes the filter criteria for that customer. To illustrate the process, you'll create a `terms` filter for a customer whose ID is `customer123` and who owns products `111`, `222`, and `333`.
+
+To encode a `terms` filter for the customer, first create a roaring bitmap for the filter. This example creates a bitmap using the [PyRoaringBitMap] library so first run `pip install pyroaring` to install the library. Then serialize the bitmap and encode it using a [Base64](https://en.wikipedia.org/wiki/Base64) encoding scheme:
+


Let's include the noun after "example".

I don't think a noun is needed because "example" refers to the overarching example that illustrated bitmap filtering.

natebower · 2024-09-12T10:45:20Z

_query-dsl/term/terms.md

+```
+{% include copy.html %}
+
+Next, index the customer filter into the `customers` index. The document ID for the filter is the customer ID for the corresponding customer (in this example, `customer123`). The `customer_filter` field contains the bitmap you generated for this customer:


"The document ID for the filter is the same as the ID for the corresponding customer"?

_query-dsl/term/terms.md

Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>

…ect#8133) * draft Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> * Doc review Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Update _query-dsl/term/terms.md Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> --------- Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Co-authored-by: Fanit Kolchina <kolchfa@amazon.com> Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Noah Staveley <noah.staveley@intel.com>

draft

7ae4989

Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com>

bowenlan-amzn requested review from kolchfa-aws, Naarcha-AWS, vagimeli, AMoo-Miki, natebower, dlvenable, stephen-crawford and epugh as code owners August 30, 2024 09:35

github-actions bot assigned kolchfa-aws Aug 30, 2024

kolchfa-aws added v2.17.0 release-notes PR: Include this PR in the automated release notes labels Aug 30, 2024

Doc review

6abce34

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

kolchfa-aws added the 1 - Backlog - DEV Developer assigned to issue is responsible for creating PR. label Aug 30, 2024

Merge branch 'main' into bitmap-filtering

4266d1b

bowenlan-amzn commented Sep 4, 2024

View reviewed changes

kolchfa-aws reviewed Sep 4, 2024

View reviewed changes

_query-dsl/term/terms.md Outdated Show resolved Hide resolved

Update _query-dsl/term/terms.md

d4f0ec1

Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>

kolchfa-aws approved these changes Sep 4, 2024

View reviewed changes

kolchfa-aws added 6 - Done but waiting to merge PR: The work is done and ready to merge 5 - Editorial review PR: Editorial review in progress and removed 1 - Backlog - DEV Developer assigned to issue is responsible for creating PR. labels Sep 4, 2024

natebower reviewed Sep 12, 2024

View reviewed changes

kolchfa-aws reviewed Sep 12, 2024

View reviewed changes

_query-dsl/term/terms.md Outdated Show resolved Hide resolved

kolchfa-aws reviewed Sep 12, 2024

View reviewed changes

_query-dsl/term/terms.md Outdated Show resolved Hide resolved

Apply suggestions from code review

729555f

Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>

kolchfa-aws merged commit 4c1e782 into opensearch-project:main Sep 13, 2024
5 checks passed

bowenlan-amzn deleted the bitmap-filtering branch September 16, 2024 22:14

tkykenmt mentioned this pull request Sep 26, 2024

[DOC] Instruction for terms query with roaring bitmap needs to be fixed #8379

Open

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Terms query can accept encoded terms input as bitmap #8133

Terms query can accept encoded terms input as bitmap #8133

bowenlan-amzn commented Aug 30, 2024 •

edited

Loading

github-actions bot commented Aug 30, 2024

kolchfa-aws commented Aug 30, 2024

bowenlan-amzn Sep 4, 2024 •

edited

Loading

kolchfa-aws left a comment

natebower left a comment

natebower Sep 12, 2024

kolchfa-aws Sep 12, 2024

natebower Sep 12, 2024

natebower Sep 12, 2024

kolchfa-aws Sep 12, 2024

natebower Sep 12, 2024

		The `terms` query can filter for multiple terms simultaneously. However, when the number of terms in the input filter increases to a large value (around 10,000), the resulting network and memory overhead can become significant, making the query inefficient. In such cases, consider encoding your large terms filter using a [roaring bitmap](https://github.com/RoaringBitmap/RoaringBitmap) for more efficient filtering.

		For example, you have two indexes: the `products` index, which contains all the products a company sells, and the `customers` index, which stores filters representing customers who own specific products.

		For each customer, you need to generate a bitmap that represents the product IDs of the products the customer owns. This bitmap effectively encodes the filter criteria for that customer. To illustrate the process, you'll create a `terms` filter for a customer whose ID is `customer123` and who owns products `111`, `222`, and `333`.

		To encode a `terms` filter for the customer, first create a roaring bitmap for the filter. This example creates a bitmap using the [PyRoaringBitMap] library so first run `pip install pyroaring` to install the library. Then serialize the bitmap and encode it using a [Base64](https://en.wikipedia.org/wiki/Base64) encoding scheme:

Terms query can accept encoded terms input as bitmap #8133

Terms query can accept encoded terms input as bitmap #8133

Conversation

bowenlan-amzn commented Aug 30, 2024 • edited Loading

Description

Issues Resolved

Version

Checklist

github-actions bot commented Aug 30, 2024

kolchfa-aws commented Aug 30, 2024

bowenlan-amzn Sep 4, 2024 • edited Loading

Choose a reason for hiding this comment

kolchfa-aws left a comment

Choose a reason for hiding this comment

natebower left a comment

Choose a reason for hiding this comment

natebower Sep 12, 2024

Choose a reason for hiding this comment

kolchfa-aws Sep 12, 2024

Choose a reason for hiding this comment

natebower Sep 12, 2024

Choose a reason for hiding this comment

natebower Sep 12, 2024

Choose a reason for hiding this comment

kolchfa-aws Sep 12, 2024

Choose a reason for hiding this comment

natebower Sep 12, 2024

Choose a reason for hiding this comment

bowenlan-amzn commented Aug 30, 2024 •

edited

Loading

bowenlan-amzn Sep 4, 2024 •

edited

Loading