[BUG][Data] Support for custom filters with heterogeneous data fields #5577

kavilla · 2023-12-06T10:35:14Z

Description

When enabling the advanced setting courier:ignoreFilterIfFieldNotInIndex Custom OpenSearch Query DSL filters could technically be applied to index patterns that map to indices that are not exactly the same. Since the custom query filter is a user input then users can really type anything that they need. Or any field that they know is present but we do not know for sure.

Therefore, we can check if the id which is the index pattern title to check if we should apply the filter or not.

⚠️ Bug impacts: 1.0-1.3., 2.0-2.11, and 3.0 ⚠️

Issues Resolved

#5581
I believe issue:
#5423

Should closed as that is expected functionality.

Screenshot

Request before:

Request after:

Testing the changes

Using the sample data web logs and created a data table.

Check List

All tests pass
- yarn test:jest
- yarn test:jest_integration
New functionality includes testing.
New functionality has been documented.
Update CHANGELOG.md
Commits are signed per the DCO using --signoff

When enabling the advanced setting `courier:ignoreFilterIfFieldNotInIndex` Custom OpenSearch Query DSL filters could technically be applied to index patterns that map to indices that are not exactly the same. Since the custom query filter is a user input then users can really type anything that they need. Or any field that they know is present but we do not know for sure. Therefore, we can check if the id which is the index pattern title to check if we should apply the filter or not. Issue resolved: https://github.com/opensearch-project/dashboards-visualizations/issues/281 I believe issue: opensearch-project#5423 Should closed as that is expected functionality. Signed-off-by: Kawika Avilla <kavilla414@gmail.com>

codecov · 2023-12-06T10:50:21Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (2e7a2ba) 66.93% compared to head (fd418fc) 66.98%.
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5577      +/-   ##
==========================================
+ Coverage   66.93%   66.98%   +0.04%     
==========================================
  Files        3293     3293              
  Lines       63285    63287       +2     
  Branches    10061    10062       +1     
==========================================
+ Hits        42362    42394      +32     
+ Misses      18459    18453       -6     
+ Partials     2464     2440      -24

Flag	Coverage Δ
Linux_1	`35.24% <0.00%> (-0.01%)`	⬇️
Linux_2	`55.21% <0.00%> (?)`
Linux_3	`43.78% <100.00%> (?)`
Linux_4	`35.34% <0.00%> (?)`
Windows_1	`35.27% <0.00%> (-0.01%)`	⬇️
Windows_2	`55.17% <0.00%> (-0.01%)`	⬇️
Windows_3	`43.80% <100.00%> (+<0.01%)`	⬆️
Windows_4	`35.34% <0.00%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

ananzh · 2023-12-06T22:21:24Z

Issue Resolve #5423

Originally when implementing the fix the historical comment caused concern about potential breaking changes. But after discussion, we decided it is more clear to consolidate the helper functions. Signed-off-by: Kawika Avilla <kavilla414@gmail.com>

src/plugins/data/common/opensearch_query/opensearch_query/filter_matches_index.ts

joshuarrrr · 2023-12-07T01:24:06Z

What do we think about backporting this to 1.3?

kavilla · 2023-12-07T01:26:30Z

What do we think about backporting this to 1.3?

i think so

kavilla · 2023-12-07T01:26:39Z

Verified newest commit locally.

…#5577) * [BUG][Data] Support for custom filters with heterogeneous data fields When enabling the advanced setting `courier:ignoreFilterIfFieldNotInIndex` Custom OpenSearch Query DSL filters could technically be applied to index patterns that map to indices that are not exactly the same. Since the custom query filter is a user input then users can really type anything that they need. Or any field that they know is present but we do not know for sure. Therefore, we can check if the id which is the index pattern title to check if we should apply the filter or not. Issue resolved: https://github.com/opensearch-project/dashboards-visualizations/issues/281 I believe issue: #5423 Should closed as that is expected functionality. Signed-off-by: Kawika Avilla <kavilla414@gmail.com> * [Cleanup] utilize the same helper function Originally when implementing the fix the historical comment caused concern about potential breaking changes. But after discussion, we decided it is more clear to consolidate the helper functions. Signed-off-by: Kawika Avilla <kavilla414@gmail.com> (cherry picked from commit 805400d) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> # Conflicts: # CHANGELOG.md

…opensearch-project#5577) * [BUG][Data] Support for custom filters with heterogeneous data fields When enabling the advanced setting `courier:ignoreFilterIfFieldNotInIndex` Custom OpenSearch Query DSL filters could technically be applied to index patterns that map to indices that are not exactly the same. Since the custom query filter is a user input then users can really type anything that they need. Or any field that they know is present but we do not know for sure. Therefore, we can check if the id which is the index pattern title to check if we should apply the filter or not. Issue resolved: https://github.com/opensearch-project/dashboards-visualizations/issues/281 I believe issue: opensearch-project#5423 Should closed as that is expected functionality. Signed-off-by: Kawika Avilla <kavilla414@gmail.com> * [Cleanup] utilize the same helper function Originally when implementing the fix the historical comment caused concern about potential breaking changes. But after discussion, we decided it is more clear to consolidate the helper functions. Signed-off-by: Kawika Avilla <kavilla414@gmail.com> Signed-off-by: Alankarsharma <alankar.sharma005@gmail.com>

…#5577) (#5585) * [BUG][Data] Support for custom filters with heterogeneous data fields When enabling the advanced setting `courier:ignoreFilterIfFieldNotInIndex` Custom OpenSearch Query DSL filters could technically be applied to index patterns that map to indices that are not exactly the same. Since the custom query filter is a user input then users can really type anything that they need. Or any field that they know is present but we do not know for sure. Therefore, we can check if the id which is the index pattern title to check if we should apply the filter or not. Issue resolved: https://github.com/opensearch-project/dashboards-visualizations/issues/281 I believe issue: #5423 Should closed as that is expected functionality. Signed-off-by: Kawika Avilla <kavilla414@gmail.com> * [Cleanup] utilize the same helper function Originally when implementing the fix the historical comment caused concern about potential breaking changes. But after discussion, we decided it is more clear to consolidate the helper functions. Signed-off-by: Kawika Avilla <kavilla414@gmail.com> (cherry picked from commit 805400d) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…#5577) (#5583) * [BUG][Data] Support for custom filters with heterogeneous data fields When enabling the advanced setting `courier:ignoreFilterIfFieldNotInIndex` Custom OpenSearch Query DSL filters could technically be applied to index patterns that map to indices that are not exactly the same. Since the custom query filter is a user input then users can really type anything that they need. Or any field that they know is present but we do not know for sure. Therefore, we can check if the id which is the index pattern title to check if we should apply the filter or not. Issue resolved: https://github.com/opensearch-project/dashboards-visualizations/issues/281 I believe issue: #5423 Should closed as that is expected functionality. Signed-off-by: Kawika Avilla <kavilla414@gmail.com> * [Cleanup] utilize the same helper function Originally when implementing the fix the historical comment caused concern about potential breaking changes. But after discussion, we decided it is more clear to consolidate the helper functions. Signed-off-by: Kawika Avilla <kavilla414@gmail.com> (cherry picked from commit 805400d) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…#5577) * [BUG][Data] Support for custom filters with heterogeneous data fields When enabling the advanced setting `courier:ignoreFilterIfFieldNotInIndex` Custom OpenSearch Query DSL filters could technically be applied to index patterns that map to indices that are not exactly the same. Since the custom query filter is a user input then users can really type anything that they need. Or any field that they know is present but we do not know for sure. Therefore, we can check if the id which is the index pattern title to check if we should apply the filter or not. Issue resolved: https://github.com/opensearch-project/dashboards-visualizations/issues/281 I believe issue: #5423 Should closed as that is expected functionality. Signed-off-by: Kawika Avilla <kavilla414@gmail.com> * [Cleanup] utilize the same helper function Originally when implementing the fix the historical comment caused concern about potential breaking changes. But after discussion, we decided it is more clear to consolidate the helper functions. Signed-off-by: Kawika Avilla <kavilla414@gmail.com> (cherry picked from commit 805400d) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> # Conflicts: # CHANGELOG.md

…#5577) (#5584) * [BUG][Data] Support for custom filters with heterogeneous data fields When enabling the advanced setting `courier:ignoreFilterIfFieldNotInIndex` Custom OpenSearch Query DSL filters could technically be applied to index patterns that map to indices that are not exactly the same. Since the custom query filter is a user input then users can really type anything that they need. Or any field that they know is present but we do not know for sure. Therefore, we can check if the id which is the index pattern title to check if we should apply the filter or not. Issue resolved: https://github.com/opensearch-project/dashboards-visualizations/issues/281 I believe issue: #5423 Should closed as that is expected functionality. Signed-off-by: Kawika Avilla <kavilla414@gmail.com> * [Cleanup] utilize the same helper function Originally when implementing the fix the historical comment caused concern about potential breaking changes. But after discussion, we decided it is more clear to consolidate the helper functions. Signed-off-by: Kawika Avilla <kavilla414@gmail.com> (cherry picked from commit 805400d) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…opensearch-project#5577) * [BUG][Data] Support for custom filters with heterogeneous data fields When enabling the advanced setting `courier:ignoreFilterIfFieldNotInIndex` Custom OpenSearch Query DSL filters could technically be applied to index patterns that map to indices that are not exactly the same. Since the custom query filter is a user input then users can really type anything that they need. Or any field that they know is present but we do not know for sure. Therefore, we can check if the id which is the index pattern title to check if we should apply the filter or not. Issue resolved: https://github.com/opensearch-project/dashboards-visualizations/issues/281 I believe issue: opensearch-project#5423 Should closed as that is expected functionality. Signed-off-by: Kawika Avilla <kavilla414@gmail.com> * [Cleanup] utilize the same helper function Originally when implementing the fix the historical comment caused concern about potential breaking changes. But after discussion, we decided it is more clear to consolidate the helper functions. Signed-off-by: Kawika Avilla <kavilla414@gmail.com> Signed-off-by: Yuanqi(Ella) Zhu <zhyuanqi@amazon.com>

…opensearch-project#5577) * [BUG][Data] Support for custom filters with heterogeneous data fields When enabling the advanced setting `courier:ignoreFilterIfFieldNotInIndex` Custom OpenSearch Query DSL filters could technically be applied to index patterns that map to indices that are not exactly the same. Since the custom query filter is a user input then users can really type anything that they need. Or any field that they know is present but we do not know for sure. Therefore, we can check if the id which is the index pattern title to check if we should apply the filter or not. Issue resolved: https://github.com/opensearch-project/dashboards-visualizations/issues/281 I believe issue: opensearch-project#5423 Should closed as that is expected functionality. Signed-off-by: Kawika Avilla <kavilla414@gmail.com> * [Cleanup] utilize the same helper function Originally when implementing the fix the historical comment caused concern about potential breaking changes. But after discussion, we decided it is more clear to consolidate the helper functions. Signed-off-by: Kawika Avilla <kavilla414@gmail.com>

kavilla requested review from ananzh, AMoo-Miki, ashwin-pc, joshuarrrr, abbyhu2000, zengyan-amazon, kristenTian, zhongnansu, manasvinibs, ZilongX, Flyingliuhub, BSFishy, curq and bandinib-amzn as code owners December 6, 2023 10:35

github-actions bot added the distinguished-contributor label Dec 6, 2023

kavilla added the backport 2.x label Dec 6, 2023

kavilla force-pushed the kavilla/custom_filter_matches_index branch from a96f8fe to 036b6f6 Compare December 6, 2023 10:40

kavilla linked an issue Dec 6, 2023 that may be closed by this pull request

[BUG]match_all query filters applied despite courier:ignoreFilterIfFieldNotInIndex setting enabled #5423

Closed

kavilla force-pushed the kavilla/custom_filter_matches_index branch from 036b6f6 to 8551434 Compare December 6, 2023 10:43

ananzh previously approved these changes Dec 6, 2023

View reviewed changes

ananzh added the v2.12.0 label Dec 6, 2023

kavilla dismissed ananzh’s stale review via fd418fc December 7, 2023 01:13

joshuarrrr approved these changes Dec 7, 2023

View reviewed changes

src/plugins/data/common/opensearch_query/opensearch_query/filter_matches_index.ts Show resolved Hide resolved

kavilla mentioned this pull request Dec 7, 2023

courier:ignoreFilterIfFieldNotInIndex filters ensure non-breaking change #5582

Open

kavilla requested a review from ananzh December 7, 2023 01:33

kavilla added backport 1.3 backport 2.11 labels Dec 7, 2023

kavilla linked an issue Dec 7, 2023 that may be closed by this pull request

[BUG] Support custom filters when ignoreFilterIfFieldNotInIndex is enabled #5581

Closed

ZilongX approved these changes Dec 7, 2023

View reviewed changes

kavilla merged commit 805400d into opensearch-project:main Dec 7, 2023
74 checks passed

opensearch-trigger-bot bot mentioned this pull request Dec 7, 2023

[Backport 1.3] [BUG][Data] Support for custom filters with heterogeneous data fields #5583

Merged

opensearch-trigger-bot bot mentioned this pull request Dec 7, 2023

[Backport 2.x] [BUG][Data] Support for custom filters with heterogeneous data fields #5584

Merged

opensearch-trigger-bot bot mentioned this pull request Dec 7, 2023

[Backport 2.11] [BUG][Data] Support for custom filters with heterogeneous data fields #5585

Merged

kavilla added the backport 2.3 label Dec 13, 2023

opensearch-trigger-bot bot mentioned this pull request Dec 13, 2023

[Backport 2.3] [BUG][Data] Support for custom filters with heterogeneous data fields #5603

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG][Data] Support for custom filters with heterogeneous data fields #5577

[BUG][Data] Support for custom filters with heterogeneous data fields #5577

kavilla commented Dec 6, 2023 •

edited

Loading

codecov bot commented Dec 6, 2023 •

edited

Loading

ananzh commented Dec 6, 2023

joshuarrrr commented Dec 7, 2023

kavilla commented Dec 7, 2023

kavilla commented Dec 7, 2023

[BUG][Data] Support for custom filters with heterogeneous data fields #5577

[BUG][Data] Support for custom filters with heterogeneous data fields #5577

Conversation

kavilla commented Dec 6, 2023 • edited Loading

Description

⚠️ Bug impacts: 1.0-1.3., 2.0-2.11, and 3.0 ⚠️

Issues Resolved

Screenshot

Testing the changes

Check List

codecov bot commented Dec 6, 2023 • edited Loading

Codecov Report

ananzh commented Dec 6, 2023

joshuarrrr commented Dec 7, 2023

kavilla commented Dec 7, 2023

kavilla commented Dec 7, 2023

kavilla commented Dec 6, 2023 •

edited

Loading

codecov bot commented Dec 6, 2023 •

edited

Loading