Rebase on main (#363)

* Update gradle version to 7.6 (#265)

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>

* Exclude lombok generated code from jacoco coverage report (#268)

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Make jacoco report to be generated faster in local (#267)

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Update dependency org.json:json to v20230227 (#273)

Co-authored-by: mend-for-github-com[bot] <50673670+mend-for-github-com[bot]@users.noreply.github.com>

* Baseline owners and maintainers (#275)

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>

* Add Auto Release Workflow (#288)

Signed-off-by: Naveen Tatikonda <navtat@amazon.com>

* Change package for Strings.hasText (#314)

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Adding release notes for 2.8 (#323)

Signed-off-by: Martin Gaievski <gaievski@amazon.com>

* Add 2.9.0 release notes (#350)

Signed-off-by: Junqiu Lei <junqiu@amazon.com>

* Update packages according to a change in OpenSearch core (#353)

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Implement creation of ip2geo feature (#257)

* Update gradle version to 7.6 (#265)

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>

* Implement creation of ip2geo feature

* Implementation of ip2geo datasource creation
* Implementation of ip2geo processor creation

Signed-off-by: Heemin Kim <heemin@amazon.com>
---------

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>
Signed-off-by: Heemin Kim <heemin@amazon.com>
Co-authored-by: Vijayan Balasubramanian <balasvij@amazon.com>

* Added unit tests with some refactoring of codes (#271)

* Add Unit tests
* Set cache true for search query
* Remove in memory cache implementation (Two way door decision)
 * Relying on search cache without custom cache
* Renamed datasource state from FAILED to CREATE_FAILED
* Renamed class name from *Helper to *Facade
* Changed updateIntervalInDays to updateInterval
* Changed value type of default update_interval from TimeValue to Long
* Read setting value from cluster settings directly

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Sync from main (#280)

* Update gradle version to 7.6 (#265)

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>

* Exclude lombok generated code from jacoco coverage report (#268)

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Make jacoco report to be generated faster in local (#267)

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Update dependency org.json:json to v20230227 (#273)

Co-authored-by: mend-for-github-com[bot] <50673670+mend-for-github-com[bot]@users.noreply.github.com>

* Baseline owners and maintainers (#275)

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>

---------

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>
Signed-off-by: Heemin Kim <heemin@amazon.com>
Co-authored-by: Vijayan Balasubramanian <balasvij@amazon.com>
Co-authored-by: mend-for-github-com[bot] <50673670+mend-for-github-com[bot]@users.noreply.github.com>

* Add datasource name validation (#281)

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Refactoring of code (#282)

1. Change variable name from datasourceName to name
2. Change variable name from id to name
3. Added helper methods in test code

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Change field name from md5 to sha256 (#285)

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Implement get datasource api (#279)

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Update index option (#284)

1. Make geodata index as hidden
2. Make geodata index as read only allow delete after creation is done
3. Refresh datasource index immediately after update

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Make some fields in manifest file as mandatory (#289)

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Create datasource index explicitly (#283)

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Add wrapper class of job scheduler lock service (#290)

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Remove all unused client attributes (#293)

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Update copyright header (#298)

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Run system index handling code with stashed thread context (#297)

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Reduce lock duration and renew the lock during update (#299)

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Implements delete datasource API (#291)

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Set User-Agent in http request (#300)

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Implement datasource update API (#292)

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Refactoring test code (#302)

Make buildGeoJSONFeatureProcessorConfig method to be more general

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Add ip2geo processor integ test for failure case (#303)

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Bug fix and refactoring of code (#305)

1. Bugfix: Ingest metadata can be null if there is no processor created
2. Refactoring: Moved private method to another class for better testing support
3. Refactoring: Set some private static final variable as public so that unit test can use it
4. Refactoring: Changed string value to static variable

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Add integration test for Ip2GeoProcessor (#306)

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Add ConcurrentModificationException (#308)

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Add integration test for UpdateDatasource API (#307)

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Bug fix on lock management and few performance improvements (#310)

* Release lock before response back to caller for update/delete API
* Release lock in background task for creation API
* Change index settings to improve indexing performance

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Change index setting from read_only_allow_delete to write (#311)

read_only_allow_delete does not block write to an index.
The disk-based shard allocator may add and remove this block automatically.
Therefore, use index.blocks.write instead.

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Fix bug in get datasource API and improve memory usage (#313)

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Change package for Strings.hasText (#314) (#317)

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Remove jitter and move index setting from DatasourceFacade to DatasourceExtension (#319)

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Do not index blank value and do not enrich null property (#320)

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Move index setting keys to constants (#321)

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Return null index name for expired data (#322)

Return null index name for expired data so that it can be deleted
by clean up process. Clean up process exclude current index from deleting.
Signed-off-by: Heemin Kim <heemin@amazon.com>

* Add new fields in datasource (#325)

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Delete index once it is expired (#326)

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Add restoring event listener (#328)

In the listener, we trigger a geoip data update

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Reverse forcemerge and refresh order (#331)

Otherwise, opensearch does not clear old segment files

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Removed parameter and settings (#332)

* Removed first_only parameter
* Removed max_concurrency and batch_size setting

first_only parameter was added as current geoip processor has it.
However, the parameter have no benefit for ip2geo processor as we don't do a sequantial search for array data but use multi search.

max_concurrency and batch_size setting is removed as these are only reveal internal implementation and could be a future blocker to improve performance later.

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Add a field in datasource for current index name (#333)

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Delete GeoIP data indices after restoring complete (#334)

We don't want to use restored GeoIP data indices. Therefore we
delete the indices once restoring process complete.

When GeoIP metadata index is restored, we create a new GeoIP data index instead.

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Use bool query for array form of IPs (#335)

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Run update/delete request in a new thread (#337)

This is not to block transport thread

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Remove IP2Geo processor validation (#336)

Cannot query index to get data to validate IP2Geo processor.
Will add validation when we decide to store some of data in cluster state metadata.

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Acquire lock sychronously (#339)

By acquiring lock asychronously, the remaining part of the code
is being run by transport thread which does not allow blocking code.
We want only single update happen in a node using single thread. However,
it cannot be acheived if I acquire lock asynchronously and pass the listener.

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Added a cache to store datasource metadata (#338)

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Changed class name and package (#341)

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Refactoring of code (#342)

1. Changed class name from Ip2GeoCache to Ip2GeoCachedDao
2. Moved the Ip2GeoCachedDao from cache to dao package

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Add geo data cache (#340)

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Add cache layer to reduce GeoIp data retrieval latency (#343)

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Use _primary in query preference and few changes (#347)

1. Use _primary preference to get datasource metadata so that it can read the latest data. RefreshPolicy.IMMEDIATE won't refresh replica shards immediately according to #346
2. Update datasource metadata index mapping
3. Move batch size from static value to setting

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Wait until GeoIP data to be replicated to all data nodes (#348)

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Update packages according to a change in OpenSearch core (#354)

* Update packages according to a change in OpenSearch core

Signed-off-by: Heemin Kim <heemin@amazon.com>

* Update packages according to a change in OpenSearch core (#353)

Signed-off-by: Heemin Kim <heemin@amazon.com>

---------

Signed-off-by: Heemin Kim <heemin@amazon.com>

---------

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>
Signed-off-by: Heemin Kim <heemin@amazon.com>
Signed-off-by: Naveen Tatikonda <navtat@amazon.com>
Signed-off-by: Martin Gaievski <gaievski@amazon.com>
Signed-off-by: Junqiu Lei <junqiu@amazon.com>
Co-authored-by: Vijayan Balasubramanian <balasvij@amazon.com>
Co-authored-by: mend-for-github-com[bot] <50673670+mend-for-github-com[bot]@users.noreply.github.com>
Co-authored-by: Naveen Tatikonda <navtat@amazon.com>
Co-authored-by: Martin Gaievski <gaievski@amazon.com>
Co-authored-by: Junqiu Lei <junqiu@amazon.com>