Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Backport 2.x] Introducing ZStd compression codec plugin (#9658) #9832

Merged
merged 1 commit into from
Sep 6, 2023
Merged

[Backport 2.x] Introducing ZStd compression codec plugin (#9658) #9832

merged 1 commit into from
Sep 6, 2023

Conversation

andrross
Copy link
Member

@andrross andrross commented Sep 6, 2023

Backports 76f1b52 from #9658 to 2.x

@github-actions
Copy link
Contributor

github-actions bot commented Sep 6, 2023

Compatibility status:

Checks if related components are compatible with change 4096817

Incompatible components

Skipped components

Compatible components

@github-actions
Copy link
Contributor

github-actions bot commented Sep 6, 2023

Gradle Check (Jenkins) Run Completed with:

@andrross andrross force-pushed the backport/backport-9658-to-2.x branch from 4096817 to 13639a8 Compare September 6, 2023 18:14
@github-actions
Copy link
Contributor

github-actions bot commented Sep 6, 2023

Gradle Check (Jenkins) Run Completed with:

@sarthakaggarwal97 @backslasht @andrross
Introducing ZStd compression codec plugin (opensearch-project#9658)
a1d85ea 
* introducing zstd compression codec plugin

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>

* Moving zstd compression codec as a plugin

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>

* introducing zstd compression codec plugin

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>

* Adding checks to EngineConfig and fixing tests

Signed-off-by: Prabhakar Sithanandam <backslasht@users.noreply.github.com>

* incorporating review comments

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>

* fixing tests

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>

* introducing zstd compression codec plugin

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>

* addressing review comments

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>

* nit fixes

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>

* implementing codec aliases

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>

* addressing review comments

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>

* review comments

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>

* moving codec aliases to custom codec

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>

* adding zstd default codec for backward compatibility

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>

* renaming to deprecated codec

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>

* incorporating review comments

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>

* nit fixes

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>

---------

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
Signed-off-by: Prabhakar Sithanandam <backslasht@users.noreply.github.com>
Signed-off-by: Andrew Ross <andrross@amazon.com>
Co-authored-by: Prabhakar Sithanandam <backslasht@users.noreply.github.com>
Co-authored-by: Andrew Ross <andrross@amazon.com>
(cherry picked from commit 76f1b52)
Signed-off-by: Andrew Ross <andrross@amazon.com>
@andrross andrross force-pushed the backport/backport-9658-to-2.x branch from 13639a8 to a1d85ea Compare September 6, 2023 18:35
@github-actions
Copy link
Contributor

github-actions bot commented Sep 6, 2023

Compatibility status:

Checks if related components are compatible with change 13639a8

Incompatible components

Incompatible components: [https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git]

@github-actions
Copy link
Contributor

github-actions bot commented Sep 6, 2023

Compatibility status:

Checks if related components are compatible with change a1d85ea

Incompatible components

Incompatible components: [https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git]

@github-actions
Copy link
Contributor

github-actions bot commented Sep 6, 2023

Gradle Check (Jenkins) Run Completed with:

  • RESULT: UNSTABLE ❕
  • TEST FAILURES:
      2 org.opensearch.index.shard.RemoteStoreRefreshListenerTests.classMethod
      1 org.opensearch.index.shard.RemoteStoreRefreshListenerTests.testReplicaPromotion

@codecov
Copy link

codecov bot commented Sep 6, 2023

Codecov Report

Merging #9832 (a1d85ea) into 2.x (0d4263e) will decrease coverage by 0.74%.
Report is 1 commits behind head on 2.x.
The diff coverage is 87.30%.

@@             Coverage Diff              @@
##                2.x    #9832      +/-   ##
============================================
- Coverage     71.32%   70.59%   -0.74%     
+ Complexity    58565    57945     -620     
============================================
  Files          4815     4820       +5     
  Lines        274976   275019      +43     
  Branches      40412    40422      +10     
============================================
- Hits         196135   194151    -1984     
- Misses        62415    64223    +1808     
- Partials      16426    16645     +219
Files Changed Coverage Δ
...customcodecs/Lucene95CustomStoredFieldsFormat.java 74.07% <ø> (ø)
.../index/codec/customcodecs/ZstdCompressionMode.java 86.66% <ø> (ø)
.../codec/customcodecs/ZstdNoDictCompressionMode.java 79.45% <ø> (ø)
.../java/org/opensearch/index/codec/CodecAliases.java 0.00% <0.00%> (ø)
.../java/org/opensearch/index/codec/CodecService.java 63.15% <ø> (+5.15%) ⬆️
.../index/codec/customcodecs/ZstdDeprecatedCodec.java 50.00% <50.00%> (ø)
...java/org/opensearch/index/engine/EngineConfig.java 94.61% <89.47%> (-1.45%) ⬇️
...h/index/codec/customcodecs/CustomCodecService.java 93.33% <93.33%> (ø)
...ch/index/codec/customcodecs/CustomCodecPlugin.java 100.00% <100.00%> (ø)
.../codec/customcodecs/CustomCodecServiceFactory.java 100.00% <100.00%> (ø)
... and 3 more

... and 517 files with indirect coverage changes

@andrross
Copy link
Member Author

andrross commented Sep 6, 2023

@reta This is ready to go. Thanks!

@reta reta merged commit 90310c3 into opensearch-project:2.x Sep 6, 2023
12 checks passed
@andrross andrross deleted the backport/backport-9658-to-2.x branch September 6, 2023 20:06
@reta reta added the backport 2.10 Backport to 2.10 branch label Sep 6, 2023
@opensearch-trigger-bot opensearch-trigger-bot bot mentioned this pull request Sep 6, 2023
Merged
opensearch-trigger-bot bot pushed a commit that referenced this pull request Sep 6, 2023
@github-actions @sarthakaggarwal97 @backslasht
Introducing ZStd compression codec plugin (#9658) (#9832)
7f0fc6e 
* introducing zstd compression codec plugin

* Moving zstd compression codec as a plugin

* introducing zstd compression codec plugin

* Adding checks to EngineConfig and fixing tests

* incorporating review comments

* fixing tests

* introducing zstd compression codec plugin

* addressing review comments

* nit fixes

* implementing codec aliases

* addressing review comments

* review comments

* moving codec aliases to custom codec

* adding zstd default codec for backward compatibility

* renaming to deprecated codec

* incorporating review comments

* nit fixes

---------

(cherry picked from commit 76f1b52)

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
Signed-off-by: Prabhakar Sithanandam <backslasht@users.noreply.github.com>
Signed-off-by: Andrew Ross <andrross@amazon.com>
Co-authored-by: Sarthak Aggarwal <sarthagg@amazon.com>
Co-authored-by: Prabhakar Sithanandam <backslasht@users.noreply.github.com>
(cherry picked from commit 90310c3)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
reta pushed a commit that referenced this pull request Sep 7, 2023
@opensearch-trigger-bot @github-actions
@sarthakaggarwal97 @backslasht
Introducing ZStd compression codec plugin (#9658) (#9832) (#9844)
55fc032 
* introducing zstd compression codec plugin

* Moving zstd compression codec as a plugin

* introducing zstd compression codec plugin

* Adding checks to EngineConfig and fixing tests

* incorporating review comments

* fixing tests

* introducing zstd compression codec plugin

* addressing review comments

* nit fixes

* implementing codec aliases

* addressing review comments

* review comments

* moving codec aliases to custom codec

* adding zstd default codec for backward compatibility

* renaming to deprecated codec

* incorporating review comments

* nit fixes

---------

(cherry picked from commit 76f1b52)






(cherry picked from commit 90310c3)

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
Signed-off-by: Prabhakar Sithanandam <backslasht@users.noreply.github.com>
Signed-off-by: Andrew Ross <andrross@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Sarthak Aggarwal <sarthagg@amazon.com>
Co-authored-by: Prabhakar Sithanandam <backslasht@users.noreply.github.com>
DarshitChanpura pushed a commit to DarshitChanpura/OpenSearch that referenced this pull request Sep 27, 2023
@opensearch-trigger-bot @github-actions
@sarthakaggarwal97 @backslasht @DarshitChanpura
Introducing ZStd compression codec plugin (opensearch-project#9658) (o…
46e9344 
…pensearch-project#9832) (opensearch-project#9844)

* introducing zstd compression codec plugin

* Moving zstd compression codec as a plugin

* introducing zstd compression codec plugin

* Adding checks to EngineConfig and fixing tests

* incorporating review comments

* fixing tests

* introducing zstd compression codec plugin

* addressing review comments

* nit fixes

* implementing codec aliases

* addressing review comments

* review comments

* moving codec aliases to custom codec

* adding zstd default codec for backward compatibility

* renaming to deprecated codec

* incorporating review comments

* nit fixes

---------

(cherry picked from commit 76f1b52)

(cherry picked from commit 90310c3)

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
Signed-off-by: Prabhakar Sithanandam <backslasht@users.noreply.github.com>
Signed-off-by: Andrew Ross <andrross@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Sarthak Aggarwal <sarthagg@amazon.com>
Co-authored-by: Prabhakar Sithanandam <backslasht@users.noreply.github.com>
DarshitChanpura pushed a commit to DarshitChanpura/OpenSearch that referenced this pull request Sep 28, 2023
@opensearch-trigger-bot @github-actions
@sarthakaggarwal97 @backslasht @DarshitChanpura
Introducing ZStd compression codec plugin (opensearch-project#9658) (o…
cace166 
…pensearch-project#9832) (opensearch-project#9844)

* introducing zstd compression codec plugin

* Moving zstd compression codec as a plugin

* introducing zstd compression codec plugin

* Adding checks to EngineConfig and fixing tests

* incorporating review comments

* fixing tests

* introducing zstd compression codec plugin

* addressing review comments

* nit fixes

* implementing codec aliases

* addressing review comments

* review comments

* moving codec aliases to custom codec

* adding zstd default codec for backward compatibility

* renaming to deprecated codec

* incorporating review comments

* nit fixes

---------

(cherry picked from commit 76f1b52)

(cherry picked from commit 90310c3)

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
Signed-off-by: Prabhakar Sithanandam <backslasht@users.noreply.github.com>
Signed-off-by: Andrew Ross <andrross@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Sarthak Aggarwal <sarthagg@amazon.com>
Co-authored-by: Prabhakar Sithanandam <backslasht@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@reta reta reta approved these changes

@anasalkouz anasalkouz Awaiting requested review from anasalkouz anasalkouz is a code owner

@Bukhtawar Bukhtawar Awaiting requested review from Bukhtawar Bukhtawar is a code owner

@CEHENKLE CEHENKLE Awaiting requested review from CEHENKLE CEHENKLE is a code owner

@dblock dblock Awaiting requested review from dblock dblock is a code owner

@gbbafna gbbafna Awaiting requested review from gbbafna gbbafna is a code owner

@setiah setiah Awaiting requested review from setiah

@kartg kartg Awaiting requested review from kartg

@kotwanikunal kotwanikunal Awaiting requested review from kotwanikunal kotwanikunal is a code owner

@mch2 mch2 Awaiting requested review from mch2 mch2 is a code owner

@nknize nknize Awaiting requested review from nknize nknize is a code owner

@owaiskazi19 owaiskazi19 Awaiting requested review from owaiskazi19 owaiskazi19 is a code owner

@peternied peternied Awaiting requested review from peternied

@Rishikesh1159 Rishikesh1159 Awaiting requested review from Rishikesh1159 Rishikesh1159 is a code owner

@ryanbogan ryanbogan Awaiting requested review from ryanbogan

@saratvemulapalli saratvemulapalli Awaiting requested review from saratvemulapalli saratvemulapalli is a code owner

@shwetathareja shwetathareja Awaiting requested review from shwetathareja shwetathareja is a code owner

@dreamer-89 dreamer-89 Awaiting requested review from dreamer-89

@tlfeng tlfeng Awaiting requested review from tlfeng

@VachaShah VachaShah Awaiting requested review from VachaShah VachaShah is a code owner

@dbwiddis dbwiddis Awaiting requested review from dbwiddis dbwiddis is a code owner

@sachinpkale sachinpkale Awaiting requested review from sachinpkale sachinpkale is a code owner

@sohami sohami Awaiting requested review from sohami sohami is a code owner

@msfroh msfroh Awaiting requested review from msfroh msfroh is a code owner

Assignees
No one assigned
Labels
backport 2.10 Backport to 2.10 branch
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@andrross @reta @sarthakaggarwal97