[Tiered Caching] Indices Request cache stalekey management (opensearc…

…h-project#12625)

* Introduce IndicesRequestCacheCleanupManager

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* using cleanup mgr to enqueue cleanups

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* readability improvements

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* update updateCleanupKeyToCountMap on new cache entry

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* create IndicesRequestCacheCleanupManager & settings and validators

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* Add IRC specific cache cleaner and remove from IndicesService

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* Unit Tests

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* Update CHANGELOG.md

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* move cachecleaner inside mgr

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* remove processCleanupKeys

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* minor cleanups

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* add updateCleanupKeyToCountMapOnCacheEviction

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* remove locks and make all methods synchronized

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* spotless

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* updateCleanupKeyToCountMapOnCacheEviction

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* Testing

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* add Reschedule back to indices service

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* rename updateStaleKeysCount to incrementStaleKeysCount

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* rename getStaleKeysCountForTesting to getStaleKeysCount

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* rename threshold to stalenessThreshold

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* check for cleanupKey.entity == null

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* use computeIfPresent with keycountmap

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* log both staleKeysInCache & Staleness in debug logs

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* Use HashMap instead of ConcurrentMap

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* Address b/w compatibility

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* remove synchronized for updateCleanupKeyToCountMapOnCacheEviction

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* make cleanCache synchronized

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* remove shouldRemoveKey

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* spotlessApply

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

---------

Signed-off-by: Kiran Prakash <awskiran@amazon.com>