add parameters to flat_object type #13853

kkewwei · 2024-05-28T10:46:53Z

Signed-off-by: kkewwei kkewwei@163.com

Description

The following open parameters setting are useful, including normalizer, docValues, ignoreAbove, nullValue, similarity, and depthlimit.

normalizer: allow lowercase, uppercase
doc_values: allow to set docValues to be false
index: allow to set index to be false
ignore_above: allow to set if the length of a field is go above certain limit then ignore the document.
similarity: allow to set similarity setting
depth_limit: allow to set maximum depth limitation to the JSON document

If there are other useful parameters, you are also welcome to give feedback or continue to add.

Related Issues

Resolves #7137

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
API changes companion pull request created.
Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)
Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2024-05-28T10:57:23Z

❌ Gradle check result for ec4d256: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-05-28T10:59:42Z

❌ Gradle check result for 25c607c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-05-28T11:27:49Z

❌ Gradle check result for 819ada8: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

kkewwei · 2024-06-25T06:58:47Z

@andrross PTAL.

andrross · 2024-06-25T15:27:48Z

@mingshl Mind taking a look here?

server/src/main/java/org/opensearch/common/xcontent/JsonToStringXContentParser.java

mingshl · 2024-06-26T16:40:27Z

server/src/main/java/org/opensearch/index/mapper/FlatObjectFieldMapper.java

+            this.indexAnalyzer = this.normalizer;
+            this.searchAnalyzer = this.normalizer;
+            if (this.normalizer == null) {
+                throw new MapperParsingException("normalizer [" + normalizer + "] is not supported in flat_object");


here sets the indexAnalyzer = this.normalizer; and searchAnalyzer = this.normalizer; but the exception mentions normalizer + "] is not supported in flat_object"? should the message be "normalizer cannot be null"?

Now we reuse the logic in keyword, please have another look.

mingshl · 2024-06-26T16:41:42Z

server/src/main/java/org/opensearch/index/mapper/FlatObjectFieldMapper.java

+
+        public void setNormalizer(String normalizer) {
+            this.normalizer = this.indexAnalyzers.getNormalizer(normalizer);
+            this.indexAnalyzer = this.normalizer;


do we need indexAnalyzer in flat object? words are not tokenized in flat object, what is this indexAnalyzer used for?

When normalizer is set, it will be used in parseValueAddFields

OpenSearch/server/src/main/java/org/opensearch/index/mapper/FlatObjectFieldMapper.java

Line 631 in c71fd4a

if (normalizer != null) {

Example in keyword type:

OpenSearch/server/src/main/java/org/opensearch/index/mapper/KeywordFieldMapper.java

Line 281 in c71fd4a

setIndexAnalyzer(normalizer);

Did I understand wrong?

server/src/main/java/org/opensearch/index/mapper/FlatObjectFieldMapper.java

mingshl · 2024-06-26T16:53:40Z

server/src/main/java/org/opensearch/index/mapper/FlatObjectFieldMapper.java

-                true,
-                new TextSearchInfo(fieldType, null, Lucene.KEYWORD_ANALYZER, Lucene.KEYWORD_ANALYZER),
+                builder.hasDocValues,
+                new TextSearchInfo(fieldType, builder.similarity, getNamedAnalyzer(builder), getNamedAnalyzer(builder)),


if we allow custom index analyzer, then flat object can be tokenized?

@mingshl, as you see above, words are not tokenized in flat object, why do we customize index analyzer? can you describe it more clearly? thank you.

server/src/main/java/org/opensearch/index/mapper/FlatObjectFieldMapper.java

mingshl · 2024-06-26T18:10:12Z

@kkewwei thanks for adding the new parameters and some yaml tests are added. I have some questions above to understand more about the logic.

In overall, UT needs to be added to validate the behavior of the new parameters, or when the new parameters go out, users would ask how to use them. It would be nice to have UTs when similarity are set, normalizer are set, doc_values are set, does that impact the search capabilities? for example, exist query and prefix query. Also some Exception cases when the parameters are null, invalid parameters are set.

Thanks!

kkewwei · 2024-06-30T04:04:35Z

@mingshl,it seems that only the null_value may affect the exist query, if here is not clear, I can temporarily rewind the logic: we should remove the sub key, when the the sub value is null.

Those unit tests add in FlatObjectFieldMapperTests.java and [106_flat_object_with_parameter.yml, including exception cases, if you have any cases, I'd be happy to add them.

kkewwei · 2024-06-30T04:12:34Z

@mingshl maybe we should first consider the case of null_value? #14069.

opensearch-trigger-bot · 2024-07-30T15:22:06Z

This PR is stalled because it has been open for 30 days with no activity.

opensearch-trigger-bot · 2024-09-08T15:22:06Z

This PR is stalled because it has been open for 30 days with no activity.

kkewwei · 2024-09-22T15:14:20Z

@msfroh I have a little doubt, please help confirm when you are free:

If we just support object value in flat_object field, do we need to support null_value?
If we do, null_value must be an object field?

github-actions · 2024-11-04T09:14:34Z

❌ Gradle check result for 46c87c3: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-11-04T09:54:01Z

❌ Gradle check result for a9f53b7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: kkewwei <kkewwei@163.com>

github-actions · 2024-11-04T11:55:40Z

❌ Gradle check result for 924f16d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

kkewwei requested review from anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, dbwiddis, dreamer-89, gbbafna, kotwanikunal, mch2, msfroh, nknize, owaiskazi19, reta, Rishikesh1159, sachinpkale, saratvemulapalli, shwetathareja, sohami, tlfeng and VachaShah as code owners May 28, 2024 10:46

github-actions bot added enhancement Enhancement or improvement to existing feature or request help wanted Extra attention is needed Search Search query, autocomplete ...etc labels May 28, 2024

kkewwei force-pushed the flat_object_para branch 2 times, most recently from 25c607c to 5bde254 Compare May 28, 2024 10:55

kkewwei force-pushed the flat_object_para branch from 819ada8 to 2607e4b Compare May 28, 2024 11:10

mingshl reviewed Jun 26, 2024

View reviewed changes

opensearch-trigger-bot bot added stalled Issues that have stalled and removed stalled Issues that have stalled labels Jul 30, 2024

kkewwei mentioned this pull request Aug 27, 2024

Fix flat_object long field error #13259

Merged

8 tasks

opensearch-trigger-bot bot added stalled Issues that have stalled and removed stalled Issues that have stalled labels Sep 8, 2024

kkewwei force-pushed the flat_object_para branch from ea2bae8 to 46c87c3 Compare November 4, 2024 09:00

kkewwei requested review from ashking94, jed326, jainankitk and linuxpi as code owners November 4, 2024 09:00

kkewwei force-pushed the flat_object_para branch from 46c87c3 to a9f53b7 Compare November 4, 2024 09:17

Add Open Parameters to Flat_object Field Type

924f16d

Signed-off-by: kkewwei <kkewwei@163.com>

kkewwei force-pushed the flat_object_para branch from a9f53b7 to 924f16d Compare November 4, 2024 11:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add parameters to flat_object type #13853

add parameters to flat_object type #13853

kkewwei commented May 28, 2024 •

edited

Loading

github-actions bot commented May 28, 2024

github-actions bot commented May 28, 2024

github-actions bot commented May 28, 2024

kkewwei commented Jun 25, 2024

andrross commented Jun 25, 2024

mingshl Jun 26, 2024

kkewwei Nov 4, 2024

mingshl Jun 26, 2024

kkewwei Jun 30, 2024 •

edited

Loading

mingshl Jun 26, 2024

kkewwei Jun 30, 2024

mingshl commented Jun 26, 2024

kkewwei commented Jun 30, 2024 •

edited

Loading

kkewwei commented Jun 30, 2024 •

edited

Loading

opensearch-trigger-bot bot commented Jul 30, 2024

opensearch-trigger-bot bot commented Sep 8, 2024

kkewwei commented Sep 22, 2024 •

edited

Loading

github-actions bot commented Nov 4, 2024

github-actions bot commented Nov 4, 2024

github-actions bot commented Nov 4, 2024

add parameters to flat_object type #13853

Are you sure you want to change the base?

add parameters to flat_object type #13853

Conversation

kkewwei commented May 28, 2024 • edited Loading

Description

Related Issues

Check List

github-actions bot commented May 28, 2024

github-actions bot commented May 28, 2024

github-actions bot commented May 28, 2024

kkewwei commented Jun 25, 2024

andrross commented Jun 25, 2024

mingshl Jun 26, 2024

Choose a reason for hiding this comment

kkewwei Nov 4, 2024

Choose a reason for hiding this comment

mingshl Jun 26, 2024

Choose a reason for hiding this comment

kkewwei Jun 30, 2024 • edited Loading

Choose a reason for hiding this comment

mingshl Jun 26, 2024

Choose a reason for hiding this comment

kkewwei Jun 30, 2024

Choose a reason for hiding this comment

mingshl commented Jun 26, 2024

kkewwei commented Jun 30, 2024 • edited Loading

kkewwei commented Jun 30, 2024 • edited Loading

opensearch-trigger-bot bot commented Jul 30, 2024

opensearch-trigger-bot bot commented Sep 8, 2024

kkewwei commented Sep 22, 2024 • edited Loading

github-actions bot commented Nov 4, 2024

github-actions bot commented Nov 4, 2024

github-actions bot commented Nov 4, 2024

kkewwei commented May 28, 2024 •

edited

Loading

kkewwei Jun 30, 2024 •

edited

Loading

kkewwei commented Jun 30, 2024 •

edited

Loading

kkewwei commented Jun 30, 2024 •

edited

Loading

kkewwei commented Sep 22, 2024 •

edited

Loading