Skip to content

Commit

Permalink
Squashed commit of the following:
Browse files Browse the repository at this point in the history 
commit 93a3132
Author: Peter Alfonsi <petealft@amazon.com>
Date:   Wed Sep 4 11:44:03 2024 -0700

    fix broken ITs

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

commit fd92c5a
Author: Marc Handalian <handalm@amazon.com>
Date:   Thu Jan 11 06:41:55 2024 -0800

    Update runTask to optionally install plugins (opensearch-project#11844)

    Signed-off-by: Marc Handalian <marc.handalian@gmail.com>

commit 4d78661
Author: Peter Alfonsi <petealft@amazon.com>
Date:   Tue Sep 3 16:02:31 2024 -0700

    Misc test fixes

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

commit 70a6935
Author: Peter Alfonsi <peter.alfonsi@gmail.com>
Date:   Fri Aug 23 10:42:05 2024 -0700

    [Bugfix] Fixes IRC NPE bug for timed-out cacheable queries (opensearch-project#15327)

    * Fix IRC timeout bug

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    * addressed Sagar's comments

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    * addressed Ankit's comments

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    * Add UT for test coverage

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    * rerun gradle

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    * tweak imports in new UT

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    * rerun gradle

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    * rerun gradle

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    * rerun gradle

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    ---------

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>
    Co-authored-by: Peter Alfonsi <petealft@amazon.com>

commit f9943f8
Author: Sagar <99425694+sgup432@users.noreply.github.com>
Date:   Wed Jul 17 19:02:05 2024 -0700

    Clear ehcache disk cache files during initialization (opensearch-project#14738)

    * Clear ehcache disk cache files during initialization

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Adding UT to fix line coverage

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Addressing comment

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Adding more Uts for better line coverage

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Throwing exception in case we fail to clear cache files during startup

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Adding more UTs

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Adding a UT for more coverage

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Fixing gradle build

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Update ehcache disk cache close() logic

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    ---------

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

commit fc2007e
Author: Peter Alfonsi <peter.alfonsi@gmail.com>
Date:   Mon Jul 1 15:54:39 2024 -0700

    [Bugfix] Fix ICacheKeySerializerTests flakiness (opensearch-project#14564)

    * Fix testInvalidInput flakiness

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    * Addressed andrross's comment

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    * rerun security check

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    ---------

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>
    Co-authored-by: Peter Alfonsi <petealft@amazon.com>

commit 1ef62e2
Author: Sagar <99425694+sgup432@users.noreply.github.com>
Date:   Wed Jun 26 11:14:42 2024 -0700

    Fix flaky test TieredSpilloverCacheTests.testComputeIfAbsentConcurrently (opensearch-project#14550)

    * Fix flaky test TieredSpilloverCacheTests.testComputeIfAbsentConcurrently

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Addressing comment

    Signed-off-by: Sagar Upadhyaya <upasagar@amazon.com>

    ---------

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>
    Signed-off-by: Sagar Upadhyaya <upasagar@amazon.com>
    Co-authored-by: Sagar Upadhyaya <upasagar@amazon.com>

commit 29ec9d6
Author: Sagar <99425694+sgup432@users.noreply.github.com>
Date:   Tue Jun 25 12:04:17 2024 -0700

    [Tiered Caching]  Moving query recomputation logic outside of write lock (opensearch-project#14187)

    * Moving query recompute out of write lock

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * [Tiered Caching] Moving query recomputation logic outside of write lock

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Adding java doc for the completable map

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Changes to call future handler only once per key

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Fixing spotless check

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Added changelog

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Addressing comments

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Fixing gradle fail

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Addressing comments to refactor unit test

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * minor UT refactor

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    ---------

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>
    Signed-off-by: Sagar <99425694+sgup432@users.noreply.github.com>
    Co-authored-by: Sagar Upadhyaya <upasagar@amazon.com>

commit 8347779
Author: Peter Alfonsi <peter.alfonsi@gmail.com>
Date:   Tue Jun 25 13:26:54 2024 -0700

    Fix flaky DefaultCacheStatsHolderTests (opensearch-project#14462)

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>
    Co-authored-by: Peter Alfonsi <petealft@amazon.com>

commit 40f868d
Author: Kiran Prakash <awskiran@amazon.com>
Date:   Thu Jun 20 14:17:23 2024 -0700

    Update IndicesRequestCacheCleanupIT.java (opensearch-project#14478)

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

commit 094a9f2
Author: Kiran Prakash <awskiran@amazon.com>
Date:   Wed Jun 19 15:36:59 2024 -0700

    Fix Indices Request Cache ITs from flaking and timing out (opensearch-project#14369)

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

commit 3a110a8
Author: Kiran Prakash <awskiran@amazon.com>
Date:   Tue Jun 18 10:18:25 2024 -0700

    [Tiered Cache] Use ConcurrentHashMap explicitly in IndicesRequestCache (opensearch-project#14409)

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

commit bcf0f59
Author: Peter Alfonsi <peter.alfonsi@gmail.com>
Date:   Fri Jun 14 13:20:51 2024 -0700

    [Bugfix] Fix TieredSpilloverCache flaky tests (opensearch-project#14333)

    * Fix flaky TSC stats tests

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    * Addressed andrross's comment

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    * fix forbidden API

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    ---------

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>
    Co-authored-by: Peter Alfonsi <petealft@amazon.com>

commit a222a03
Author: Kiran Prakash <awskiran@amazon.com>
Date:   Wed Jun 12 18:30:06 2024 -0700

    [Tiered Caching] [Bug Fix] Use concurrentMap instead of HashMap to fix Concurrent Modification Exception (opensearch-project#14221)

    * use concurrentmap

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update CHANGELOG.md

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * revert feature flags

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * changelog to releaselog

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * use concurrentmap

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update CHANGELOG.md

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * revert feature flags

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * changelog to releaselog

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * revert the test removal

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * revert the conflict resolutions

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    ---------

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

commit f85cd00
Author: Sagar <99425694+sgup432@users.noreply.github.com>
Date:   Wed Jun 12 08:22:38 2024 -0700

    Fix ShardNotFoundException during request cache clean up (opensearch-project#14219)

    * Fix for ShardNotFoundException during request cache clean up

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Added changelog

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Fix forbidden gradle check

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    ---------

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

commit 406ae78
Author: Sagar <99425694+sgup432@users.noreply.github.com>
Date:   Thu Jun 6 15:02:00 2024 -0700

    [Caching] Move cache removal notifications outside lru lock (opensearch-project#14017)

    ---------

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

commit c5fd7fe
Author: Peter Alfonsi <petealft@amazon.com>
Date:   Tue Sep 3 14:38:26 2024 -0700

    Change IRC UT to correctly create shard

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

commit 9c4d4af
Author: Peter Alfonsi <peter.alfonsi@gmail.com>
Date:   Tue Jun 4 16:09:18 2024 -0700

    [Tiered Caching] Additional ITs for cache stats  (opensearch-project#13655)

    * Adds cache clear IT

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    Cleaned up logic for cache stats ITs

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    Adds more tests around tiered spillover cache

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    Fixed cache stats behavior for overall /_nodes/stats call

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    cleanup

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    Fixed folder structure

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    Addressed Sagar's comments

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    Addressed Ankit's comments

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    Break horrifyingly long test case into many shorter cases

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    Added unsupported operation exception to TSC stats holder incrementEvictions()

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    Addressed Sorabh's comments

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    * rerun assemble

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    * rerun gradle

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    * rerun gradle

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    * rerun gradle

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    ---------

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>
    Co-authored-by: Peter Alfonsi <petealft@amazon.com>

commit be97e90
Author: Peter Alfonsi <peter.alfonsi@gmail.com>
Date:   Thu May 23 12:02:22 2024 -0700

    [Bugfix] [Tiered Caching] Fixes issues when integrating tiered cache with disk cache (opensearch-project#13784)

    ---------

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>
    Co-authored-by: Peter Alfonsi <petealft@amazon.com>

commit 60ee715
Author: Kiran Prakash <awskiran@amazon.com>
Date:   Mon May 20 12:52:03 2024 -0700

    Update IndicesRequestCacheIT.java (opensearch-project#13678)

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

commit 51f6c1b
Author: Sagar <99425694+sgup432@users.noreply.github.com>
Date:   Wed May 8 09:15:36 2024 -0700

    Fix IndicesRequestCache clean up logic (opensearch-project#13597)

    Signed-off-by: Sagar Upadhyaya <upasagar@amazon.com>
    Co-authored-by: Sagar Upadhyaya <upasagar@amazon.com>

commit 49e2701
Author: Sagar <99425694+sgup432@users.noreply.github.com>
Date:   Mon May 6 17:25:45 2024 -0700

    Fix negative requestStats memory_size issue (opensearch-project#13553)

    This solves the bug where RequestStats memory_size metric was going negative in certain scenarios
    as reported in the issue. It turns out that the issue occurs when an indexShard is deleted and then
    reallocated on the same node. So whenever stale entries from older shard are deleted, those are
    accounted for the new shard which has the same shardId.

    ---------

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

commit a5c5675
Author: Peter Alfonsi <peter.alfonsi@gmail.com>
Date:   Thu May 2 17:29:07 2024 -0700

    [Tiered Caching] Adds stats implementation for TieredSpilloverCache (opensearch-project#13236)

    Stats rework part 4 of 4
    ---------

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>
    Co-authored-by: Peter Alfonsi <petealft@amazon.com>

commit 46944cb
Author: peteralfonsi <peter.alfonsi@gmail.com>
Date:   Tue Apr 30 10:59:03 2024 -0700

    Fix flaky test CacheStatsAPIIndicesRequestCacheIT.testNullLevels() (opensearch-project#13457)

    * Fix flaky test

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    * Initialize CommonStatsFlags with empty array for levels

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    * Fixes tests using incorrect null levels

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    * rerun

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    ---------

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>
    Co-authored-by: Peter Alfonsi <petealft@amazon.com>

commit 71fcad4
Author: Kiran Prakash <awskiran@amazon.com>
Date:   Mon Apr 29 18:41:33 2024 -0700

    Fix Flaky test IndicesRequestCacheIT.testStaleKeysCleanupWithMultipleIndices (opensearch-project#13453)

    * Update IndicesRequestCacheIT.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCacheIT.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    ---------

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

commit f60339f
Author: peteralfonsi <peter.alfonsi@gmail.com>
Date:   Mon Apr 29 22:37:05 2024 -0700

    [Tiered Caching] Bump versions for serialization in new cache stats API (opensearch-project#13460)

    ---------

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>
    Co-authored-by: Peter Alfonsi <petealft@amazon.com>

commit 88718a2
Author: peteralfonsi <peter.alfonsi@gmail.com>
Date:   Mon Apr 29 14:47:52 2024 -0700

    [Tiered Caching] Expose new cache stats API (opensearch-project#13237)

    Step 3 out of 4

    ---------

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>
    Co-authored-by: Peter Alfonsi <petealft@amazon.com>

commit 5e97082
Author: peteralfonsi <peter.alfonsi@gmail.com>
Date:   Sun Apr 28 08:15:18 2024 -0700

    [Tiered Caching] Gate CacheStatsHolder logic behind FeatureFlags.PLUGGABLE_CACHE setting (opensearch-project#13238)

    Stats rework step 2 of 4

    ---------

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>
    Co-authored-by: Peter Alfonsi <petealft@amazon.com>

commit ed881f7
Author: Kiran Prakash <awskiran@amazon.com>
Date:   Fri Apr 26 11:52:31 2024 -0700

    [Tiered Caching] Make Indices Request Cache Stale Key Mgmt Threshold setting dynamic (opensearch-project#12941)

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update ClusterSettings.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCacheIT.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * spotless

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCacheIT.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * some refactoring

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * address existing tests

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * UTs

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update CHANGELOG.md

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * ITs

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * spotless

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * refactor

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCacheIT.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCacheIT.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCacheIT.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCacheIT.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update CHANGELOG.md

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update CHANGELOG.md

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCacheIT.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCacheIT.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * resolve conflicts

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * address code comments

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * address code comments

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCacheIT.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * rename tests

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCacheIT.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCacheIT.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * resolve conflicts

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * code comments

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCacheIT.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    ---------

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

commit 424ccc9
Author: Sagar <99425694+sgup432@users.noreply.github.com>
Date:   Thu Apr 25 22:09:42 2024 -0700

    [Tiered Caching] Expose a dynamic setting to disable/enable disk cache (opensearch-project#13373)

    * [Tiered Caching] Expose a dynamic setting to disable/enable disk cache

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Putting tiered cache settings behind feature flag

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Adding a changelog

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Addressing Sorabh's comments

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Putting new setting behind feature flag

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    ---------

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>
    Signed-off-by: Sagar <99425694+sgup432@users.noreply.github.com>

commit acbbb2f
Author: Kiran Prakash <awskiran@amazon.com>
Date:   Wed Apr 24 14:50:39 2024 -0700

    [Tiered Caching] Bug fix for IndicesRequestCache StaleKey management (opensearch-project#13070)

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update CHANGELOG.md

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * revert

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * revert

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * code comments only

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * docs changes

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update CHANGELOG.md

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * revert catching AlreadyClosedException

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * assert

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * conflicts

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * address comments

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * address conflicts

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * spotless apply

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * address comments

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * update code comments

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * address bug & add tests

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    ---------

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

commit f1d2e72
Author: Sagar <99425694+sgup432@users.noreply.github.com>
Date:   Mon Apr 15 10:43:22 2024 -0800

    [Tiered Caching] Ehcache Disk cache IT  (opensearch-project#12904)

    * Ehcache IT tests

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Adding some logs to print key/value size

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Add ehcache related invalidation IT

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Remvoing unnecessary IndicesRequestCache IT

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Indentation fix

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Added tests around expiration time and invalidation

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    ---------

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

commit b0de406
Author: peteralfonsi <peter.alfonsi@gmail.com>
Date:   Fri Apr 12 16:09:06 2024 -0700

    [Tiered Caching] Stats rework (1/3): Interfaces and implementations for individual tiers (opensearch-project#12531)

    As part of tiered caching stats, changes the common ICache interface to use ICacheKey as its key. This key contains
    dimensions (for example, shard ID, index name, or tier) that can be used to aggregate stats. Also changes the
    CacheStats interface to store the necessary cache stats, and to support getting stats either as a total or aggregated by
    these dimensions.

    Integrates these changes with OpenSearchOnHeapCache and EhcacheDiskCache. The stats implementation for the
    TieredSpilloverCache will be in a followup PR.

    ---------

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>
    Co-authored-by: Peter Alfonsi <petealft@amazon.com>

commit 40a8d34
Author: Sagar <99425694+sgup432@users.noreply.github.com>
Date:   Thu Apr 11 14:49:03 2024 -0800

    [Tiered Caching] Make took time policy dynamic and add additional integ tests (opensearch-project#13063)

    ---------

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>
    Signed-off-by: Sagar <99425694+sgup432@users.noreply.github.com>

commit 6f88777
Author: Sagar <99425694+sgup432@users.noreply.github.com>
Date:   Tue Mar 19 19:41:59 2024 -0700

    Fixing ehcache flaky test (opensearch-project#12764)

    * Fixing ehcache flaky test

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Adding a ehcache issue reference for thread leak issue

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Updating comment

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    ---------

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

commit f69c089
Author: Sagar <99425694+sgup432@users.noreply.github.com>
Date:   Mon Mar 18 22:20:52 2024 -0700

    [Tiered Caching] Clear up disk cache(ehcache) files during node shutdown (opensearch-project#12734)

    * Adding logic to clear up the disk cache files during close()
    * Adding logic to update entries count after invalidateAll()
    * Removing unneeded system log statement
    * Added comment in test for readability
    * Fixing issue where we were sending compacted byte[] array to ehcache but calculating size with padded byte[]

    ---------

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>
    Signed-off-by: Sagar <99425694+sgup432@users.noreply.github.com>

commit bf6488d
Author: Sagar <99425694+sgup432@users.noreply.github.com>
Date:   Mon Mar 18 22:17:14 2024 -0700

    [Tiered Caching] Fix test testComputeIfAbsentWithFactoryBasedCacheCreation (opensearch-project#12700)

    ---------

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

commit 0fc1f40
Author: peteralfonsi <peter.alfonsi@gmail.com>
Date:   Mon Mar 18 13:39:35 2024 -0700

    [Tiered Caching] Serializers for ehcache (opensearch-project#12709)

    Adds serializers and integrates them into ehcache disk cache

    ---------

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>
    Co-authored-by: Peter Alfonsi <petealft@amazon.com>

commit a45be0f
Author: Kiran Prakash <awskiran@amazon.com>
Date:   Sat Mar 16 17:12:27 2024 -0700

    [Tiered Caching] Indices Request cache stalekey management (opensearch-project#12625)

    * Introduce IndicesRequestCacheCleanupManager

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * using cleanup mgr to enqueue cleanups

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * readability improvements

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * update updateCleanupKeyToCountMap on new cache entry

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * create IndicesRequestCacheCleanupManager & settings and validators

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Add IRC specific cache cleaner and remove from IndicesService

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Unit Tests

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Update CHANGELOG.md

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * move cachecleaner inside mgr

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * remove processCleanupKeys

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * minor cleanups

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * add updateCleanupKeyToCountMapOnCacheEviction

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * remove locks and make all methods synchronized

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * spotless

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * updateCleanupKeyToCountMapOnCacheEviction

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Testing

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * add Reschedule back to indices service

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * rename updateStaleKeysCount to incrementStaleKeysCount

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * rename getStaleKeysCountForTesting to getStaleKeysCount

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * rename threshold to stalenessThreshold

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * check for cleanupKey.entity == null

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * use computeIfPresent with keycountmap

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * log both staleKeysInCache & Staleness in debug logs

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Use HashMap instead of ConcurrentMap

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * Address b/w compatibility

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * remove synchronized for updateCleanupKeyToCountMapOnCacheEviction

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * make cleanCache synchronized

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * remove shouldRemoveKey

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    * spotlessApply

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

    ---------

    Signed-off-by: Kiran Prakash <awskiran@amazon.com>

commit bad412c
Author: Sagar <99425694+sgup432@users.noreply.github.com>
Date:   Fri Mar 15 16:53:54 2024 -0700

    [Tiered caching] Supporting removal function on EhcacheDiskCache iterator (opensearch-project#12653)

    * [Tiered caching] Supporting removal function on EhcacheDiskCache iterator

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Minor refactoring in unit test

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    ---------

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

commit 15c4afa
Author: peteralfonsi <peter.alfonsi@gmail.com>
Date:   Fri Mar 15 15:36:22 2024 -0700

    [Tiered Caching] Cache tier policies (opensearch-project#12542)

    * Adds policy interface and took time policy impl

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    * Changes IndicesService to write a CachePolicyInfoWrapper before the QSR

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    * Moved took time logic from QSR to IndicesService

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    * spotlessApply

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    * Addressed ansjcy's comments

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    * Partial rebase on most recent changes

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    * Integrated policies with new TSC changes

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    * Reverted unintended change to idea/vcs.xml

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    * javadocs

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    * github actions

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    * Set default threshold value to 10 ms

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    * Addressed Sorabh's comments

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    * Addressed Sorabh's second round of comments

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    * Set cachedQueryParser in IRC

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    * Addressed Sorabh's comments besides dynamic setting

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    * Removed dynamic setting, misc comments

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    * Added changelog entry

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    * Added missing javadoc

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    * Fixed failed gradle run

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    * Added setting validation test

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    * rerun gradle for flaky IT

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    * javadocs

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

    ---------

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>
    Co-authored-by: Peter Alfonsi <petealft@amazon.com>

commit 45fce15
Author: Sagar <99425694+sgup432@users.noreply.github.com>
Date:   Thu Mar 14 13:00:24 2024 -0700

    [Tiered Caching] Fixing flaky tiered cache test (opensearch-project#12650)

    * Fixing flaky tiered cache test

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Removing unnecessary comment

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Removing unused variable

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    ---------

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

commit bb66699
Author: Peter Alfonsi <petealft@amazon.com>
Date:   Fri Aug 30 11:36:05 2024 -0700

    Attempt to fix broken ITs

    Signed-off-by: Peter Alfonsi <petealft@amazon.com>

commit 58170bf
Author: Sagar <99425694+sgup432@users.noreply.github.com>
Date:   Mon Mar 11 14:12:55 2024 -0700

    [Tiered caching] Integrating IndicesRequestCache with CacheService controlled by a feature flag (opensearch-project#12533)

    * Adding changelog
    * Fixing gradle build issue
    * Fixing CacheService test
    * Adding UT in IndicesRequestCache with feature flag for more coverage
    * Updating changelog and renaming feature flag setting
    * Moving feature flag setting handling logic to CacheService by maintaining backward compatibility
    * Fixing broken UTs

    ---------

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>
    Signed-off-by: Sagar <99425694+sgup432@users.noreply.github.com>

commit 80d2130
Author: Sagar <99425694+sgup432@users.noreply.github.com>
Date:   Fri Mar 1 11:10:28 2024 -0800

    [Tiered caching] Introducing cache plugins and exposing Ehcache as one of the pluggable disk cache option (opensearch-project#11874)

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>
    Signed-off-by: Sagar <99425694+sgup432@users.noreply.github.com>

commit f372c2c
Author: Sagar <99425694+sgup432@users.noreply.github.com>
Date:   Wed Jan 10 17:06:29 2024 -0800

    [Tiered Caching] Enable serialization of IndicesRequestCache.Key (opensearch-project#10275)

    ---------

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>
    Signed-off-by: Sagar <99425694+sgup432@users.noreply.github.com>
    Co-authored-by: Kiran Prakash <awskiran@amazon.com>

commit b880c0b
Author: Sagar <99425694+sgup432@users.noreply.github.com>
Date:   Tue Jan 9 10:57:52 2024 -0800

    [Tiered caching] Framework changes (opensearch-project#10753)

    * [Tiered caching] Framework changes

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Added javadoc for new files/packages

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Added changelog

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Fixing javadoc warnings

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Addressing comments

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Addressing additional minor comments

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Moving non null check to builder for OS onHeapCache

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Adding package-info for new packages

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Removing service and adding different cache interfaces along with event listener support

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Fixing gradle missingDoc issue

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Changing listener logic, removing tiered cache integration with IRC

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Adding opensearch.internal tag for LoadAwareCacheLoader

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Fixing thread safety issue

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Remove compute function and event listener logic change for TieredCache

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Making Cache.compute function private

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Adding javadoc and more test for cache.put

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Adding write locks to refresh API as well

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Removing unwanted EventType class and refactoring one UT

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    * Removing TieredCache interface

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

    ---------

    Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>
    Signed-off-by: Sagar <99425694+sgup432@users.noreply.github.com>

Signed-off-by: Peter Alfonsi <petealft@amazon.com>
  • Loading branch information
Peter Alfonsi committed Sep 4, 2024
1 parent a5cc554 commit 60f3710
Show file tree
Hide file tree
Showing 119 changed files with 15,642 additions and 529 deletions.
8 changes: 7 additions & 1 deletion DEVELOPER_GUIDE.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,12 @@ Run OpenSearch using `gradlew run`.
./gradlew run
```

[Plugins](plugins/) may be installed by passing a `-PinstalledPlugins` property:

```bash
./gradlew run -PinstalledPlugins="['plugin1', 'plugin2']"
```

That will build OpenSearch and start it, writing its log above Gradle's status message. We log a lot of stuff on startup, specifically these lines tell you that OpenSearch is ready.

```
Expand Down Expand Up @@ -578,7 +584,7 @@ explicitly marked by an annotation should not be extended by external implementa
any time. The `@DeprecatedApi` annotation could also be added to any classes annotated with `@PublicApi` (or documented as `@opensearch.api`) or their methods that
are either changed (with replacement) or planned to be removed across major versions.

The APIs which are designated to be public but have not been stabilized yet should be marked with `@ExperimentalApi` (or documented as `@opensearch.experimental`)
The APIs which are designated to be public but have not been stabilized yet should be marked with `@ExperimentalApi` (or documented as `@opensearch.experimental`)
annotation. The presence of this annotation signals that API may change at any time (major, minor or even patch releases). In general, the classes annotated with
`@PublicApi` may expose other classes or methods annotated with `@ExperimentalApi`, in such cases the backward compatibility guarantees would not apply to latter
(see please [Experimental Development](#experimental-development) for more details).
Expand Down
3 changes: 3 additions & 0 deletions distribution/src/config/opensearch.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -121,3 +121,6 @@ ${path.logs}
# index searcher threadpool.
#
#opensearch.experimental.feature.concurrent_segment_search.enabled: false
# Gates the functionality of enabling Opensearch to use pluggable caches with respective store names via setting.
#
#opensearch.experimental.feature.pluggable.caching.enabled: false
6 changes: 6 additions & 0 deletions gradle/run.gradle
View file
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,12 @@ apply plugin: 'opensearch.testclusters'
testClusters {
runTask {
testDistribution = 'archive'
if (findProperty("installedPlugins")) {
installedPlugins = Eval.me(installedPlugins)
for (String p : installedPlugins) {
plugin('plugins:'.concat(p))
}
}
}
}

Expand Down
5 changes: 5 additions & 0 deletions libs/core/src/main/java/org/opensearch/core/common/bytes/BytesReference.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,11 @@ static byte[] toBytes(BytesReference reference) {
return ArrayUtil.copyOfSubArray(bytesRef.bytes, bytesRef.offset, bytesRef.offset + bytesRef.length);
}

static byte[] toBytesWithoutCompact(BytesReference reference) {
final BytesRef bytesRef = reference.toBytesRef();
return bytesRef.bytes;
}

/**
* Returns an array of byte buffers from the given BytesReference.
*/
Expand Down
40 changes: 39 additions & 1 deletion libs/core/src/main/java/org/opensearch/core/common/io/stream/StreamInput.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@
import java.util.Arrays;
import java.util.Collection;
import java.util.Collections;
import java.util.Comparator;
import java.util.Date;
import java.util.EnumSet;
import java.util.HashMap;
Expand All @@ -89,6 +90,8 @@
import java.util.Locale;
import java.util.Map;
import java.util.Set;
import java.util.SortedMap;
import java.util.TreeMap;
import java.util.concurrent.TimeUnit;
import java.util.function.IntFunction;

Expand Down Expand Up @@ -641,12 +644,47 @@ public <K, V> Map<K, V> readMap(Writeable.Reader<K> keyReader, Writeable.Reader<
return Collections.emptyMap();
}
Map<K, V> map = new HashMap<>(size);
readIntoMap(keyReader, valueReader, map, size);
return map;
}

/**
* Read a serialized map into a SortedMap using the default ordering for the keys. If the result is empty it might be immutable.
*/
public <K extends Comparable<K>, V> SortedMap<K, V> readOrderedMap(Writeable.Reader<K> keyReader, Writeable.Reader<V> valueReader)
throws IOException {
return readOrderedMap(keyReader, valueReader, null);
}

/**
* Read a serialized map into a SortedMap, specifying a Comparator for the keys. If the result is empty it might be immutable.
*/
public <K extends Comparable<K>, V> SortedMap<K, V> readOrderedMap(
Writeable.Reader<K> keyReader,
Writeable.Reader<V> valueReader,
@Nullable Comparator<K> keyComparator
) throws IOException {
int size = readArraySize();
if (size == 0) {
return Collections.emptySortedMap();
}
SortedMap<K, V> sortedMap;
if (keyComparator == null) {
sortedMap = new TreeMap<>();
} else {
sortedMap = new TreeMap<>(keyComparator);
}
readIntoMap(keyReader, valueReader, sortedMap, size);
return sortedMap;
}

private <K, V> void readIntoMap(Writeable.Reader<K> keyReader, Writeable.Reader<V> valueReader, Map<K, V> map, int size)
throws IOException {
for (int i = 0; i < size; i++) {
K key = keyReader.read(this);
V value = valueReader.read(this);
map.put(key, value);
}
return map;
}

/**
Expand Down
7 changes: 7 additions & 0 deletions libs/core/src/main/java/org/opensearch/core/index/shard/ShardId.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@

package org.opensearch.core.index.shard;

import org.apache.lucene.util.RamUsageEstimator;
import org.opensearch.common.annotation.PublicApi;
import org.opensearch.core.common.Strings;
import org.opensearch.core.common.io.stream.StreamInput;
Expand All @@ -55,6 +56,8 @@ public class ShardId implements Comparable<ShardId>, ToXContentFragment, Writeab
private final int shardId;
private final int hashCode;

private final static long BASE_RAM_BYTES_USED = RamUsageEstimator.shallowSizeOfInstance(ShardId.class);

/**
* Constructs a new shard id.
* @param index the index name
Expand Down Expand Up @@ -88,6 +91,10 @@ public ShardId(StreamInput in) throws IOException {
hashCode = computeHashCode();
}

public long getBaseRamBytesUsed() {
return BASE_RAM_BYTES_USED;
}

/**
* Writes this shard id to a stream.
* @param out the stream to write to
Expand Down
19 changes: 19 additions & 0 deletions modules/cache-common/build.gradle
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
/*
* SPDX-License-Identifier: Apache-2.0
*
* The OpenSearch Contributors require contributions made to
* this file be licensed under the Apache-2.0 license or a
* compatible open source license.
*/

apply plugin: 'opensearch.internal-cluster-test'

opensearchplugin {
description 'Module for caches which are optional and do not require additional security permission'
classname 'org.opensearch.cache.common.tier.TieredSpilloverCachePlugin'
}

test {
// TODO: Adding permission in plugin-security.policy doesn't seem to work.
systemProperty 'tests.security.manager', 'false'
}
Loading

0 comments on commit 60f3710

Please sign in to comment.