[Draft] Search Query Categorizor initial skeleton using QueryBuilderVisitor #10255

deshsidd · 2023-09-27T18:45:20Z

Initial skeleton changes
Using QueryBuilderVisitor to visit all the nodes of the querybuilder tree
The tree referred to here is the tree of querybuilders for the incoming search query with the topLevelQueryBuilder as the root.
Integration with Metric Framework : Add Telemetry metrics framework #10241
Integration with QueryShapeVisitor to log the shape of the queries : https://gist.github.com/msfroh/74aa3fee52f4074c5e7b8d85f76e88ab
Need to also figure out all the types of queries we want to increment counters for

github-actions · 2023-09-27T18:50:20Z

Compatibility status:

Checks if related components are compatible with change 3338d4f

Incompatible components

Skipped components

Compatible components

github-actions · 2023-09-27T18:57:21Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/26397/
CommitID: d1a1f2c
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

msfroh · 2023-10-02T18:35:06Z

The sample implementation doesn't tell us whether given query clause is required or not, or if it's involved in scoring. These have major implications for performance.

Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com>

…ch-project#10347) * Update multipart download path to write to temp files. This change updates ReadContextListener to first write parts to a temp location until all parts have been received. Signed-off-by: Marc Handalian <handalm@amazon.com> * Suppress forbidden IOUtils.fsync Signed-off-by: Marc Handalian <handalm@amazon.com> * Remove unnecessary logging format Signed-off-by: Marc Handalian <handalm@amazon.com> * sync directory after file rename Signed-off-by: Marc Handalian <handalm@amazon.com> * Remove flaky threadpool terminate test Signed-off-by: Marc Handalian <handalm@amazon.com> --------- Signed-off-by: Marc Handalian <handalm@amazon.com>

…nsearch-project#9569) --------- Signed-off-by: Bhumika Saini <sabhumik@amazon.com>

…oject#10326) Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

…ject#10205) Bumps org.apache.commons:commons-compress from 1.23.0 to 1.24.0. --- updated-dependencies: - dependency-name: org.apache.commons:commons-compress dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Signed-off-by: Gagan Juneja <gjjuneja@amazon.com>

…h-project#10016) * fix stale remote cluster uuid state not purged from remote Signed-off-by: bansvaru <bansvaru@amazon.com> * fix tests Signed-off-by: bansvaru <bansvaru@amazon.com> * use new limit parameter Signed-off-by: bansvaru <bansvaru@amazon.com> * minor refactoring Signed-off-by: bansvaru <bansvaru@amazon.com> * delete index metadata files before manifest file Signed-off-by: bansvaru <bansvaru@amazon.com> * add basic UT Signed-off-by: bansvaru <bansvaru@amazon.com> * delete all data related to a cluster uuid in a single call Signed-off-by: bansvaru <bansvaru@amazon.com> * fix git diff Signed-off-by: bansvaru <bansvaru@amazon.com> * remove unreferenced code Signed-off-by: bansvaru <bansvaru@amazon.com> * fix spa Signed-off-by: bansvaru <bansvaru@amazon.com> --------- Signed-off-by: bansvaru <bansvaru@amazon.com>

…n/tools/plugin-cli (opensearch-project#10297) * Bump org.bouncycastle:bc-fips in /distribution/tools/plugin-cli Bumps org.bouncycastle:bc-fips from 1.0.2.3 to 1.0.2.4. --- updated-dependencies: - dependency-name: org.bouncycastle:bc-fips dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

Signed-off-by: Bhumika Saini <sabhumik@amazon.com>

* Race condition fix for datetime optimization Signed-off-by: Prabhat Sharma <ptsharma@amazon.com> * Changed JavaDateTimeFormatter caching of parser from MRU(most recently used) to a simple last used formatter Signed-off-by: Prabhat Sharma <ptsharma@amazon.com> --------- Signed-off-by: Prabhat Sharma <ptsharma@amazon.com> Co-authored-by: Prabhat Sharma <ptsharma@amazon.com>

…ata fields in ingest simulate API (opensearch-project#10101) * Fix class_cast_exception when passing int to _version and other metadata fields in ingest simulate API Signed-off-by: Gao Binlong <gbinlong@amazon.com> * modify change log Signed-off-by: Gao Binlong <gbinlong@amazon.com> * Add more tests Signed-off-by: Gao Binlong <gbinlong@amazon.com> --------- Signed-off-by: Gao Binlong <gbinlong@amazon.com> Signed-off-by: Daniel (dB.) Doubrovkine <dblock@amazon.com> Co-authored-by: Daniel (dB.) Doubrovkine <dblock@amazon.com>

…ases (opensearch-project#10370) * Segment Replication - Fix ShardLockObtained error during corruption cases This change fixes a bug where shards could not be recreated locally after corruption. This occured because the store was not decref'd to 0 if the commit on close would fail with a corruption exception. Signed-off-by: Marc Handalian <handalm@amazon.com> * Remove exra logs Signed-off-by: Marc Handalian <handalm@amazon.com> * Remove flaky assertion on store refcount Signed-off-by: Marc Handalian <handalm@amazon.com> * Remove flaky test. Signed-off-by: Marc Handalian <handalm@amazon.com> * PR Feedback. Remove hacky handling of corruption when fetching metadata. This will now check for store corruption when replication has failed and fail the shard accordingly. This commit also fixes logging in NRTReplicationEngine. Signed-off-by: Marc Handalian <handalm@amazon.com> * Fix unit test. Signed-off-by: Marc Handalian <handalm@amazon.com> * Fix test failure testSegRepSucceedsOnPreviousCopiedFiles. This test broke because we invoked target.indexShard on a closed replicationTarget. In these cases we can assume the store is not corrupt. Signed-off-by: Marc Handalian <handalm@amazon.com> * spotless Signed-off-by: Marc Handalian <handalm@amazon.com> * Revert flaky IT Signed-off-by: Marc Handalian <handalm@amazon.com> * Fix flakiness failure by expecting RTE when check index fails. Signed-off-by: Marc Handalian <handalm@amazon.com> * reintroduce ITs and use recoveries API instead of waiting on shard state. Signed-off-by: Marc Handalian <handalm@amazon.com> * Fix edge case where flush failures would not get reported as corruption. Signed-off-by: Marc Handalian <handalm@amazon.com> --------- Signed-off-by: Marc Handalian <handalm@amazon.com>

Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>

…arch-project#10402) * Fix stats reporting for multistream downloads. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * rename tracker to fileTransferTracker. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> --------- Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

… allowed in base64UUID url charset (opensearch-project#10379) * Refactor Remote Store Metadata Lock Manager Utils Signed-off-by: Harish Bhakuni <hbhakuni@amazon.com> * Address PR Comments Signed-off-by: Harish Bhakuni <hbhakuni@amazon.com> * Address PR Comments Signed-off-by: Harish Bhakuni <hbhakuni@amazon.com> * Update Changelog entry Signed-off-by: Harish Bhakuni <hbhakuni@amazon.com> * Update Changelog entry Signed-off-by: Harish Bhakuni <hbhakuni@amazon.com> * Unmute testDeleteShallowCopySnapshot test Signed-off-by: Harish Bhakuni <hbhakuni@amazon.com> --------- Signed-off-by: Harish Bhakuni <hbhakuni@amazon.com> Co-authored-by: Harish Bhakuni <hbhakuni@amazon.com>

…ression (opensearch-project#10261)

…eft on device' (opensearch-project#10409) Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

…ettings during s3 plugin reload (opensearch-project#10452) Signed-off-by: vikasvb90 <vikasvb@amazon.com>

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

…25.1 in /plugins/repository-gcs (opensearch-project#10298) * Bump com.google.api.grpc:proto-google-common-protos Bumps [com.google.api.grpc:proto-google-common-protos](https://github.com/googleapis/sdk-platform-java) from 2.25.0 to 2.25.1. - [Release notes](https://github.com/googleapis/sdk-platform-java/releases) - [Changelog](https://github.com/googleapis/sdk-platform-java/blob/main/CHANGELOG.md) - [Commits](https://github.com/googleapis/sdk-platform-java/commits) --- updated-dependencies: - dependency-name: com.google.api.grpc:proto-google-common-protos dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

…project#10437) * DiversifiedSamplerIT.testNestedSamples * QueryProfilePhaseTests.testMaxScore * QueryProfilePhaseTests.testCollapseQuerySearchResults * HighlighterSearchIT.testHighlightQueryRewriteDatesWithNow * FieldCapabilitiesIT.testWithIndexFilter * QueryProfilePhaseTests.testDisableTopScoreCollection Signed-off-by: Jay Deng <jayd0104@gmail.com> Signed-off-by: Jay Deng <dengjay@amazon.com>

…25.1 in /plugins/repository-gcs (opensearch-project#10298) (opensearch-project#10461) * Bump com.google.api.grpc:proto-google-common-protos Bumps [com.google.api.grpc:proto-google-common-protos](https://github.com/googleapis/sdk-platform-java) from 2.25.0 to 2.25.1. - [Release notes](https://github.com/googleapis/sdk-platform-java/releases) - [Changelog](https://github.com/googleapis/sdk-platform-java/blob/main/CHANGELOG.md) - [Commits](https://github.com/googleapis/sdk-platform-java/commits) --- updated-dependencies: - dependency-name: com.google.api.grpc:proto-google-common-protos dependency-type: direct:production update-type: version-update:semver-patch ... * Updating SHAs * Update changelog --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

…ch-project#10480) Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>

…erIT (opensearch-project#10464) Signed-off-by: Ashish Singh <ssashish@amazon.com>

…rTests (opensearch-project#10490) Signed-off-by: Ashish Singh <ssashish@amazon.com>

…ad & store recovery (opensearch-project#10449) --------- Signed-off-by: Ashish Singh <ssashish@amazon.com>

Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com>

github-actions · 2023-10-09T08:53:45Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/27426/
CommitID: 3338d4f
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

deshsidd changed the title ~~Search Query Categorizor initial skeleton using QueryBuilderVisitor~~ [Draft] Search Query Categorizor initial skeleton using QueryBuilderVisitor Sep 27, 2023

Search Query Categorizor initial skeleton using QueryBuilderVisitor

fadba4c

Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com>

deshsidd force-pushed the sid/search-query-categorizor branch from d1a1f2c to fadba4c Compare October 4, 2023 05:57

mch2 and others added 27 commits October 9, 2023 01:10

[Remote Store] Add support to reload repository metadata inplace (ope…

d1d3027

…nsearch-project#9569) --------- Signed-off-by: Bhumika Saini <sabhumik@amazon.com>

Adding cancellation start time in client side TaskInfo (opensearch-pr…

6995db5

…oject#10326) Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

Add Telemetry metrics framework (opensearch-project#10241)

ad110b9

Signed-off-by: Gagan Juneja <gjjuneja@amazon.com>

Update TODO for updating public documentation (opensearch-project#10408)

d333dd6

Signed-off-by: Bhumika Saini <sabhumik@amazon.com>

Fix error handling for future completion (opensearch-project#10406)

3676eff

Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>

Allow customization of netty channel handles before and during decomp…

747d108

…ression (opensearch-project#10261)

Disable build cache since Github Action runners fail with 'no space l…

9c1910e

…eft on device' (opensearch-project#10409) Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

Fix to include static settings in addition to dynamic repo metadata s…

179e53b

…ettings during s3 plugin reload (opensearch-project#10452) Signed-off-by: vikasvb90 <vikasvb@amazon.com>

Update Gradle to 8.4 (opensearch-project#10138)

802d11a

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

[Remote Store] Using hash of node id in metadata file names (opensear…

312c536

…ch-project#10480) Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>

Fix flaky testRemoteRefreshRetryOnFailure in RemoteStoreRefreshListen…

c3440ba

…erIT (opensearch-project#10464) Signed-off-by: Ashish Singh <ssashish@amazon.com>

Fix failing testGetPrimaryTermAndGeneration in TranslogTransferManage…

17c32af

…rTests (opensearch-project#10490) Signed-off-by: Ashish Singh <ssashish@amazon.com>

Fix bugs causing red indexes with remote indexes during translog uplo…

d8666f0

…ad & store recovery (opensearch-project#10449) --------- Signed-off-by: Ashish Singh <ssashish@amazon.com>

Update changelog

9e58f1b

Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com>

Integrate metrics framework, refactor code and update tests

3338d4f

Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com>

deshsidd closed this Oct 9, 2023

BrewTestBot mentioned this pull request Feb 21, 2024

opensearch 2.12.0 Homebrew/homebrew-core#163463

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Draft] Search Query Categorizor initial skeleton using QueryBuilderVisitor #10255

[Draft] Search Query Categorizor initial skeleton using QueryBuilderVisitor #10255

deshsidd commented Sep 27, 2023 •

edited

Loading

github-actions bot commented Sep 27, 2023 •

edited

Loading

github-actions bot commented Sep 27, 2023

msfroh commented Oct 2, 2023

github-actions bot commented Oct 9, 2023

[Draft] Search Query Categorizor initial skeleton using QueryBuilderVisitor #10255

[Draft] Search Query Categorizor initial skeleton using QueryBuilderVisitor #10255

Conversation

deshsidd commented Sep 27, 2023 • edited Loading

github-actions bot commented Sep 27, 2023 • edited Loading

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Sep 27, 2023

Gradle Check (Jenkins) Run Completed with:

msfroh commented Oct 2, 2023

github-actions bot commented Oct 9, 2023

Gradle Check (Jenkins) Run Completed with:

deshsidd commented Sep 27, 2023 •

edited

Loading

github-actions bot commented Sep 27, 2023 •

edited

Loading