Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Star tree] Star tree file formats refactoring and fixing offset bug #15975

Merged
merged 2 commits into from
Sep 19, 2024
Merged

[Star tree] Star tree file formats refactoring and fixing offset bug #15975

merged 2 commits into from
Sep 19, 2024
+281 −209

Conversation

bharath-techie
Copy link
Contributor

@bharath-techie bharath-techie commented Sep 18, 2024 •
edited
Loading

Description

This PR removes the exposure of DocIdSetIterator in the file formats flow.

So, in order to prevent exposing DISI directly as part of star tree file format readers , this change contains a simple wrapper iterator that abstracts the DISI , and exposes relevant methods tied to star tree to retrieve the values.

Apart from this also fixing a bug where file size greater than Integer.MaxValue is not handled correctly.

Related Issues

#15974 - issue for the bug found with offsets

Check List

  • Functionality includes testing.
  • API changes companion pull request created, if applicable.
  • Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for f6d3ca9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@bharath-techie bharath-techie force-pushed the starrefactor branch 2 times, most recently from 67e8e7e to 14a7575 Compare September 18, 2024 10:35
@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for 67e8e7e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for 14a7575: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@opensearch-ci-bot opensearch-ci-bot mentioned this pull request Sep 18, 2024
Open
@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for bbc8e2f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@sandeshkr419 sandeshkr419 reopened this Sep 18, 2024
@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for bbc8e2f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@sandeshkr419 sandeshkr419 reopened this Sep 18, 2024
@sandeshkr419 sandeshkr419 added the backport 2.x Backport to 2.x branch label Sep 18, 2024
@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for bbc8e2f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@sandeshkr419
Copy link
Contributor

Flaky test: #15944

@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for bbc8e2f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@sandeshkr419
Copy link
Contributor

sandeshkr419 commented Sep 18, 2024 •
edited
Loading

The PR branch also has this 2.18.0 tag in bwcversion: https://github.com/bharath-techie/OpenSearch/blob/starrefactor/.ci/bwcVersions

Still bwc is failing:

Execution failed for task ':distribution:bwc:staged:buildBwcLinuxTar'.
> Building 2.17.0 didn't generate expected file /var/jenkins/workspace/gradle-check/search/distribution/bwc/staged/build/bwc/checkout-2.17/distribution/archives/linux-tar/build/distributions/opensearch-min-2.17.0-SNAPSHOT-linux-x64.tar.gz

@bharath-techie Can you rebase and force push your changes once?

@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for cb324b6: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@bharath-techie
Copy link
Contributor Author

#14324 flaky test

@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for b4f38ea: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

This was referenced Sep 16, 2024
Open
Open
Open
@github-actions GitHub Actions
Copy link
Contributor

❕ Gradle check result for b4f38ea: UNSTABLE

  • TEST FAILURES:
      1 org.opensearch.remotestore.RemoteStoreStatsIT.testDownloadStatsCorrectnessSinglePrimaryMultipleReplicaShards

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

@codecov Codecov
Copy link

codecov bot commented Sep 19, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 77.04918% with 14 lines in your changes missing coverage. Please review.

Project coverage is 71.92%. Comparing base (3937ccb) to head (b4f38ea).
Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
...be/startree/utils/SequentialDocValuesIterator.java 68.75% 2 Missing and 3 partials ⚠️
...cube/startree/builder/StarTreeDocsFileManager.java 50.00% 1 Missing and 2 partials ⚠️
...eindex/datacube/startree/index/StarTreeValues.java 76.92% 1 Missing and 2 partials ⚠️
...tartree/utils/iterator/StarTreeValuesIterator.java 57.14% 3 Missing ⚠️
Additional details and impacted files 
@@             Coverage Diff              @@
##               main   #15975      +/-   ##
============================================
+ Coverage     71.90%   71.92%   +0.02%     
+ Complexity    64392    64351      -41     
============================================
  Files          5278     5280       +2     
  Lines        300877   300870       -7     
  Branches      43478    43476       -2     
============================================
+ Hits         216351   216412      +61     
+ Misses        66747    66655      -92     
- Partials      17779    17803      +24

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@sachinpkale sachinpkale merged commit 80ff07e into opensearch-project:main Sep 19, 2024
61 of 64 checks passed
opensearch-trigger-bot bot pushed a commit that referenced this pull request Sep 19, 2024
@github-actions
star tree file formats refactoring and fixing offset bug (#15975)
732aa12 
Signed-off-by: Bharathwaj G <bharath78910@gmail.com>
(cherry picked from commit 80ff07e)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@opensearch-trigger-bot opensearch-trigger-bot bot mentioned this pull request Sep 19, 2024
Open
@opensearch-ci-bot opensearch-ci-bot mentioned this pull request Sep 9, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@msfroh msfroh msfroh approved these changes

@sachinpkale sachinpkale sachinpkale approved these changes

@dbwiddis dbwiddis dbwiddis approved these changes

@sandeshkr419 sandeshkr419 sandeshkr419 approved these changes

@sarthakaggarwal97 sarthakaggarwal97 sarthakaggarwal97 approved these changes

@mgodwan mgodwan mgodwan approved these changes

@anasalkouz anasalkouz Awaiting requested review from anasalkouz anasalkouz is a code owner

@andrross andrross Awaiting requested review from andrross andrross is a code owner

@ashking94 ashking94 Awaiting requested review from ashking94 ashking94 is a code owner

@Bukhtawar Bukhtawar Awaiting requested review from Bukhtawar Bukhtawar is a code owner

@CEHENKLE CEHENKLE Awaiting requested review from CEHENKLE CEHENKLE is a code owner

@dblock dblock Awaiting requested review from dblock dblock is a code owner

@gbbafna gbbafna Awaiting requested review from gbbafna gbbafna is a code owner

@jed326 jed326 Awaiting requested review from jed326 jed326 is a code owner

@kotwanikunal kotwanikunal Awaiting requested review from kotwanikunal kotwanikunal is a code owner

@mch2 mch2 Awaiting requested review from mch2 mch2 is a code owner

@nknize nknize Awaiting requested review from nknize nknize is a code owner

@owaiskazi19 owaiskazi19 Awaiting requested review from owaiskazi19 owaiskazi19 is a code owner

@reta reta Awaiting requested review from reta reta is a code owner

@Rishikesh1159 Rishikesh1159 Awaiting requested review from Rishikesh1159 Rishikesh1159 is a code owner

@saratvemulapalli saratvemulapalli Awaiting requested review from saratvemulapalli saratvemulapalli is a code owner

@shwetathareja shwetathareja Awaiting requested review from shwetathareja shwetathareja is a code owner

@sohami sohami Awaiting requested review from sohami sohami is a code owner

@VachaShah VachaShah Awaiting requested review from VachaShah VachaShah is a code owner

Assignees
No one assigned
Labels
backport 2.x Backport to 2.x branch skip-changelog
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@bharath-techie @sandeshkr419 @msfroh @sachinpkale @dbwiddis @sarthakaggarwal97 @mgodwan