Add experimental SIMD implementation of B-tree to round down dates #11194

ketanv3 · 2023-11-14T10:19:56Z

Description

In the date histogram aggregation, the timestamp value from each hit must be rounded down to the nearest interval (year, quarter, month, week, day, etc.) as defined in the search request. This rounded-down timestamp serves as the bucket key to aggregate results.

This PR aims to improve the performance of rounding-down dates using SIMD.
More details can be found here: #10392

Related Issues

Resolves #10392

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)
~~Public documentation issue/PR created~~

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-11-14T10:23:13Z

Compatibility status:

Checks if related components are compatible with change 88b45a3

Incompatible components

Incompatible components: [https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/geospatial.git]

github-actions · 2023-11-14T11:02:58Z

❌ Gradle check result for 923022b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2023-11-14T11:37:57Z

❕ Gradle check result for d03fc44: UNSTABLE

TEST FAILURES:

      1 org.opensearch.search.SearchWeightedRoutingIT.testMultiGetWithNetworkDisruption_FailOpenEnabled

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov · 2023-11-14T11:38:38Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (904c9a9) 71.44% compared to head (88b45a3) 71.38%.

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #11194      +/-   ##
============================================
- Coverage     71.44%   71.38%   -0.07%     
+ Complexity    59397    59345      -52     
============================================
  Files          4923     4923              
  Lines        279178   279178              
  Branches      40581    40581              
============================================
- Hits         199470   199283     -187     
- Misses        63064    63274     +210     
+ Partials      16644    16621      -23

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

github-actions · 2023-11-14T13:54:45Z

❕ Gradle check result for a67c3d9: UNSTABLE

TEST FAILURES:

      1 org.opensearch.search.SearchWeightedRoutingIT.testMultiGetWithNetworkDisruption_FailOpenEnabled

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-11-15T13:32:20Z

✅ Gradle check result for bc42b6b: SUCCESS

github-actions · 2023-11-15T20:39:06Z

✅ Gradle check result for ee1e2a2: SUCCESS

Signed-off-by: Ketan Verma <ketan9495@gmail.com>

github-actions · 2023-11-15T23:42:12Z

✅ Gradle check result for 4432ec6: SUCCESS

github-actions · 2023-11-15T23:54:19Z

✅ Gradle check result for 550422c: SUCCESS

github-actions · 2023-12-18T12:58:40Z

❕ Gradle check result for 0edb311: UNSTABLE

TEST FAILURES:

      1 org.opensearch.remotestore.RemoteIndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

libs/common/src/test/java/org/opensearch/common/round/RoundableTests.java

libs/common/src/main/java20/org/opensearch/common/round/RoundableFactory.java

This reverts commit 2e82d0a. Signed-off-by: Ketan Verma <ketan9495@gmail.com>

…on of unit-tests Signed-off-by: Ketan Verma <ketan9495@gmail.com>

github-actions · 2023-12-18T16:16:18Z

❕ Gradle check result for 5336b6d: UNSTABLE

TEST FAILURES:

      1 org.opensearch.search.SearchWeightedRoutingIT.testMultiGetWithNetworkDisruption_FailOpenEnabled

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-12-18T16:53:41Z

❕ Gradle check result for 42eac9a: UNSTABLE

TEST FAILURES:

      2 org.opensearch.remotestore.RemoteIndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

backslasht

Thanks @ketanv3 for making this changes!

libs/common/src/main/java20/org/opensearch/common/round/RoundableFactory.java

libs/common/src/main/java20/org/opensearch/common/round/BtreeSearcher.java

ketanv3 · 2023-12-21T15:40:58Z

@andrross Are you still looking at this PR?

andrross · 2024-01-18T16:43:47Z

@reta Looks like I missed this one :(

Are you still good with this? I've rebased against main so it should be good to merge once the check is green.

github-actions · 2024-01-18T17:26:20Z

❕ Gradle check result for 88b45a3: UNSTABLE

TEST FAILURES:

      1 org.opensearch.remotestore.RemoteIndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

…11194) * Add experimental SIMD implementation of B-tree to round down dates Signed-off-by: Ketan Verma <ketan9495@gmail.com> * Use system properties in favor of feature flags to remove dependency on the server module Signed-off-by: Ketan Verma <ketan9495@gmail.com> * Removed Java 20 test sources to simplify builds Signed-off-by: Ketan Verma <ketan9495@gmail.com> * Remove the use of forbidden APIs in unit-tests Signed-off-by: Ketan Verma <ketan9495@gmail.com> * Migrate to the recommended usage for custom test task classpath Signed-off-by: Ketan Verma <ketan9495@gmail.com> * Switch benchmarks module to multi-release one Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Add JMH annotation processing for JDK-20+ sources Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Make JMH annotations consistent across sources Signed-off-by: Ketan Verma <ketan9495@gmail.com> * Improve execution of Roundable unit-tests Signed-off-by: Ketan Verma <ketan9495@gmail.com> * Revert "Improve execution of Roundable unit-tests" This reverts commit 2e82d0a. Signed-off-by: Ketan Verma <ketan9495@gmail.com> * Add 'forced' as a possible feature flag value to simplify the execution of unit-tests Signed-off-by: Ketan Verma <ketan9495@gmail.com> --------- Signed-off-by: Ketan Verma <ketan9495@gmail.com> Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Co-authored-by: Andriy Redko <andriy.redko@aiven.io> Co-authored-by: Andrew Ross <andrross@amazon.com> (cherry picked from commit b424aaf) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…11194) (#11924) * Add experimental SIMD implementation of B-tree to round down dates * Use system properties in favor of feature flags to remove dependency on the server module * Removed Java 20 test sources to simplify builds * Remove the use of forbidden APIs in unit-tests * Migrate to the recommended usage for custom test task classpath * Switch benchmarks module to multi-release one * Add JMH annotation processing for JDK-20+ sources * Make JMH annotations consistent across sources * Improve execution of Roundable unit-tests * Revert "Improve execution of Roundable unit-tests" This reverts commit 2e82d0a. * Add 'forced' as a possible feature flag value to simplify the execution of unit-tests --------- (cherry picked from commit b424aaf) Signed-off-by: Ketan Verma <ketan9495@gmail.com> Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Andriy Redko <andriy.redko@aiven.io> Co-authored-by: Andrew Ross <andrross@amazon.com>

…pensearch-project#11194) * Add experimental SIMD implementation of B-tree to round down dates Signed-off-by: Ketan Verma <ketan9495@gmail.com> * Use system properties in favor of feature flags to remove dependency on the server module Signed-off-by: Ketan Verma <ketan9495@gmail.com> * Removed Java 20 test sources to simplify builds Signed-off-by: Ketan Verma <ketan9495@gmail.com> * Remove the use of forbidden APIs in unit-tests Signed-off-by: Ketan Verma <ketan9495@gmail.com> * Migrate to the recommended usage for custom test task classpath Signed-off-by: Ketan Verma <ketan9495@gmail.com> * Switch benchmarks module to multi-release one Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Add JMH annotation processing for JDK-20+ sources Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Make JMH annotations consistent across sources Signed-off-by: Ketan Verma <ketan9495@gmail.com> * Improve execution of Roundable unit-tests Signed-off-by: Ketan Verma <ketan9495@gmail.com> * Revert "Improve execution of Roundable unit-tests" This reverts commit 2e82d0a. Signed-off-by: Ketan Verma <ketan9495@gmail.com> * Add 'forced' as a possible feature flag value to simplify the execution of unit-tests Signed-off-by: Ketan Verma <ketan9495@gmail.com> --------- Signed-off-by: Ketan Verma <ketan9495@gmail.com> Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Co-authored-by: Andriy Redko <andriy.redko@aiven.io> Co-authored-by: Andrew Ross <andrross@amazon.com>

…pensearch-project#11194) * Add experimental SIMD implementation of B-tree to round down dates Signed-off-by: Ketan Verma <ketan9495@gmail.com> * Use system properties in favor of feature flags to remove dependency on the server module Signed-off-by: Ketan Verma <ketan9495@gmail.com> * Removed Java 20 test sources to simplify builds Signed-off-by: Ketan Verma <ketan9495@gmail.com> * Remove the use of forbidden APIs in unit-tests Signed-off-by: Ketan Verma <ketan9495@gmail.com> * Migrate to the recommended usage for custom test task classpath Signed-off-by: Ketan Verma <ketan9495@gmail.com> * Switch benchmarks module to multi-release one Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Add JMH annotation processing for JDK-20+ sources Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Make JMH annotations consistent across sources Signed-off-by: Ketan Verma <ketan9495@gmail.com> * Improve execution of Roundable unit-tests Signed-off-by: Ketan Verma <ketan9495@gmail.com> * Revert "Improve execution of Roundable unit-tests" This reverts commit 2e82d0a. Signed-off-by: Ketan Verma <ketan9495@gmail.com> * Add 'forced' as a possible feature flag value to simplify the execution of unit-tests Signed-off-by: Ketan Verma <ketan9495@gmail.com> --------- Signed-off-by: Ketan Verma <ketan9495@gmail.com> Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Co-authored-by: Andriy Redko <andriy.redko@aiven.io> Co-authored-by: Andrew Ross <andrross@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

github-actions bot added the enhancement Enhancement or improvement to existing feature or request label Nov 14, 2023

ketanv3 changed the title ~~Add vectorized (SIMD) implementation of B-tree to round down dates~~ [DRAFT] Add vectorized (SIMD) implementation of B-tree to round down dates Nov 14, 2023

ketanv3 force-pushed the simd/rounding branch from 923022b to d03fc44 Compare November 14, 2023 10:52

ketanv3 force-pushed the simd/rounding branch from d03fc44 to a67c3d9 Compare November 14, 2023 13:04

ketanv3 mentioned this pull request Nov 15, 2023

Refactor common parts from the Rounding class into a separate 'round' package #11023

Merged

8 tasks

ketanv3 force-pushed the simd/rounding branch from bc42b6b to ee1e2a2 Compare November 15, 2023 19:54

ketanv3 force-pushed the simd/rounding branch from ee1e2a2 to 4432ec6 Compare November 15, 2023 22:55

ketanv3 changed the title ~~[DRAFT] Add vectorized (SIMD) implementation of B-tree to round down dates~~ [DRAFT] Add experimental SIMD implementation of B-tree to round down dates Nov 15, 2023

Add experimental SIMD implementation of B-tree to round down dates

550422c

Signed-off-by: Ketan Verma <ketan9495@gmail.com>

ketanv3 force-pushed the simd/rounding branch from 4432ec6 to 550422c Compare November 15, 2023 23:03

ketanv3 marked this pull request as ready for review November 16, 2023 05:27

ketanv3 requested review from abbashus, adnapibar, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, dbwiddis, dreamer-89 and gbbafna as code owners November 16, 2023 05:27

reta reviewed Dec 18, 2023

View reviewed changes

libs/common/src/test/java/org/opensearch/common/round/RoundableTests.java Outdated Show resolved Hide resolved

reta self-requested a review December 18, 2023 14:30

reta reviewed Dec 18, 2023

View reviewed changes

libs/common/src/main/java20/org/opensearch/common/round/RoundableFactory.java Show resolved Hide resolved

reta approved these changes Dec 18, 2023

View reviewed changes

ketanv3 added 2 commits December 18, 2023 21:09

Revert "Improve execution of Roundable unit-tests"

ef85c47

This reverts commit 2e82d0a. Signed-off-by: Ketan Verma <ketan9495@gmail.com>

Add 'forced' as a possible feature flag value to simplify the executi…

42eac9a

…on of unit-tests Signed-off-by: Ketan Verma <ketan9495@gmail.com>

ketanv3 force-pushed the simd/rounding branch from 5336b6d to 42eac9a Compare December 18, 2023 15:41

backslasht approved these changes Dec 19, 2023

View reviewed changes

libs/common/src/main/java20/org/opensearch/common/round/RoundableFactory.java Show resolved Hide resolved

libs/common/src/main/java20/org/opensearch/common/round/BtreeSearcher.java Show resolved Hide resolved

getsaurabh02 mentioned this pull request Jan 5, 2024

Add performance improvement blog opensearch-project/project-website#2522

Merged

1 task

Merge branch 'main' into simd/rounding

88b45a3

andrross approved these changes Jan 18, 2024

View reviewed changes

reta merged commit b424aaf into opensearch-project:main Jan 18, 2024
30 checks passed

reta added v3.0.0 Issues and PRs related to version 3.0.0 v2.12.0 Issues and PRs related to version 2.12.0 backport 2.x Backport to 2.x branch labels Jan 18, 2024

opensearch-trigger-bot bot mentioned this pull request Jan 18, 2024

[Backport 2.x] Add experimental SIMD implementation of B-tree to round down dates #11924

Merged

reta mentioned this pull request Jan 18, 2024

[BUG] The non-core plugin test cases fail due to the use of JDK incubator features #11937

Closed

BrewTestBot mentioned this pull request Feb 21, 2024

opensearch 2.12.0 Homebrew/homebrew-core#163463

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add experimental SIMD implementation of B-tree to round down dates #11194

Add experimental SIMD implementation of B-tree to round down dates #11194

ketanv3 commented Nov 14, 2023 •

edited

Loading

github-actions bot commented Nov 14, 2023 •

edited

Loading

github-actions bot commented Nov 14, 2023

github-actions bot commented Nov 14, 2023

codecov bot commented Nov 14, 2023 •

edited

Loading

github-actions bot commented Nov 14, 2023

github-actions bot commented Nov 15, 2023

github-actions bot commented Nov 15, 2023

github-actions bot commented Nov 15, 2023

github-actions bot commented Nov 15, 2023

github-actions bot commented Dec 18, 2023

github-actions bot commented Dec 18, 2023

github-actions bot commented Dec 18, 2023

backslasht left a comment

ketanv3 commented Dec 21, 2023

andrross commented Jan 18, 2024

github-actions bot commented Jan 18, 2024

Add experimental SIMD implementation of B-tree to round down dates #11194

Add experimental SIMD implementation of B-tree to round down dates #11194

Conversation

ketanv3 commented Nov 14, 2023 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Nov 14, 2023 • edited Loading

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Nov 14, 2023

github-actions bot commented Nov 14, 2023

codecov bot commented Nov 14, 2023 • edited Loading

Codecov Report

github-actions bot commented Nov 14, 2023

github-actions bot commented Nov 15, 2023

github-actions bot commented Nov 15, 2023

github-actions bot commented Nov 15, 2023

github-actions bot commented Nov 15, 2023

github-actions bot commented Dec 18, 2023

github-actions bot commented Dec 18, 2023

github-actions bot commented Dec 18, 2023

backslasht left a comment

Choose a reason for hiding this comment

ketanv3 commented Dec 21, 2023

andrross commented Jan 18, 2024

github-actions bot commented Jan 18, 2024

ketanv3 commented Nov 14, 2023 •

edited

Loading

github-actions bot commented Nov 14, 2023 •

edited

Loading

codecov bot commented Nov 14, 2023 •

edited

Loading