[Extensions] Synchronize opensearch-plugin-job-details with JobDetailsService map #299

joshpalis · 2023-01-10T20:28:12Z

Description

Adds JobDetailsService as an indexOperationListener for index .opensearch-plugin-job-details. This allows us to update the internal JobDetails map with the entries indexed to the job details meta data index. Additionally, the indicesToListen set is updated with the job index names post-index, configuring onIndexModule to add the JobSweeper as an indexOperationListener once an extension creates their job index

Issues Resolved

Part of opensearch-project/opensearch-sdk-java#309

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…etadata index with internal job details m and indicesToListen set. Signed-off-by: Joshua Palis <jpalis@amazon.com>

vibrantvarun · 2023-01-10T20:39:03Z

@joshpalis Once the sanity test case for the workflow is added in the PR. LGTM!

…indexToJobDetails Signed-off-by: Joshua Palis <jpalis@amazon.com>

vibrantvarun · 2023-01-10T21:40:44Z

src/main/java/org/opensearch/jobscheduler/utils/JobDetailsService.java

    }

    public boolean jobDetailsIndexExist() {
        return clusterService.state().routingTable().hasIndex(JOB_DETAILS_INDEX_NAME);
    }

+    public ConcurrentMap<String, JobDetails> getIndexToJobDetails() {
+        return this.indexToJobDetails;


@joshpalis you can Implement the return of object of indexToJobdetails in below manner.

private static class InstanceHolder {
private static final YourObject instance = new YourObject();
}

public static YourObject getInstance() {
return InstanceHolder.instance;
}
This solution takes advantage of the Java memory model's guarantees about class initialization to ensure thread safety. Each class can only be loaded once, and it will only be loaded when it is needed. That means that the first time getInstance is called, InstanceHolder will be loaded and instance will be created, and since this is controlled by ClassLoaders, no additional synchronization is necessary.

In the case above it would look like

JobDetailsService.indexToJobDetails

Also rename method from getIndexToJobDetailsMap and make method static.

Sure I'll go ahead and add this, I'll keep the method name getIndexToJobDetails, appending Map isn't necessary

Signed-off-by: Joshua Palis <jpalis@amazon.com>

joshpalis · 2023-01-11T00:23:12Z

BWC tests are failing since JS main BWC tests use JS 2.5.0, which is no longer wire-compatible with JS 3.x. Currently waiting for OpenSearch 2.6.0-SNAPSHOT to be available in sonatype before merging #297. Upon merging, I'll update the JS main BWC tests to use 2.6.0 #304. This should resolve the failing checks

Signed-off-by: Joshua Palis <jpalis@amazon.com>

dbwiddis

LGTM (mostly). Several readability suggestions.

src/test/java/org/opensearch/jobscheduler/utils/JobDetailsServiceIT.java

dbwiddis · 2023-01-11T19:45:11Z

src/test/java/org/opensearch/jobscheduler/utils/JobDetailsServiceIT.java

+            }, exception -> { fail(exception.getMessage()); })
+        );
+
+        inProgressFuture.get(JobDetailsService.TIME_OUT_FOR_REQUEST, TimeUnit.SECONDS);


(Readability/Consistency) If we time out I assume the test fails with an exception. That's fine and expected, but seems to be different than line 190 and 192 where you explicitly fail() on exceptions. Would be more readable with consistent treatment of these.

Also consider many developers look at the tests to see how they should implement, so it should likely match a "real" execution workflow.

Suggest lines 190 and 192 be completeExceptionally() and you have a single try/catch around the get with a fail() in the catch block.

src/main/java/org/opensearch/jobscheduler/JobSchedulerPlugin.java

src/main/java/org/opensearch/jobscheduler/utils/JobDetailsService.java

Signed-off-by: Joshua Palis <jpalis@amazon.com>

vibrantvarun · 2023-01-12T17:28:50Z

LGTM!

dbwiddis

LGTM!

saratvemulapalli · 2023-01-12T22:20:33Z

src/main/java/org/opensearch/jobscheduler/JobSchedulerPlugin.java

@@ -105,7 +105,8 @@ public Collection<Object> createComponents(
            threadPool,
            xContentRegistry,
            this.scheduler,
-            this.lockService
+            this.lockService,
+            this.jobDetailsService


Trying to understand, why do we need jobDetailsService to createComponents?

JobDetailsService needs access to the node client to make requests and the ClusterService for the index routing table. These are both pulled from createComponents.

saratvemulapalli · 2023-01-12T22:25:07Z

src/main/java/org/opensearch/jobscheduler/JobSchedulerPlugin.java

@@ -151,6 +152,10 @@ public List<ExecutorBuilder<?>> getExecutorBuilders(Settings settings) {

    @Override
    public void onIndexModule(IndexModule indexModule) {
+        if (indexModule.getIndex().getName().equals(JobDetailsService.JOB_DETAILS_INDEX_NAME)) {


Follow the same pattern of adding it to indicesToListen?
Probably a little more concise code.

saratvemulapalli · 2023-01-12T22:34:33Z

src/main/java/org/opensearch/jobscheduler/rest/RestGetJobTypeAction.java

@@ -73,16 +73,16 @@ protected RestChannelConsumer prepareRequest(RestRequest restRequest, NodeClient
        final JobDetails[] jobDetailsResponseHolder = new JobDetails[1];

        String jobType = getJobTypeRequest.getJobType();
-        String extensionId = getJobTypeRequest.getExtensionId();
+        String extensionUniqueId = getJobTypeRequest.getExtensionUniqueId();


Not a question on this PR, but do we really need two different Rest APIs/Actions to get JobType, and get JobIndex?
We could just use the same API and reduce the duplication.

Please refer to this discussion : opensearch-project/anomaly-detection#761 (comment)

saratvemulapalli · 2023-01-13T00:35:37Z

src/main/java/org/opensearch/jobscheduler/sweeper/JobSweeper.java

@@ -103,7 +105,8 @@ public JobSweeper(
        NamedXContentRegistry registry,
        Map<String, ScheduledJobProvider> indexToProviders,
        JobScheduler scheduler,
-        LockService lockService
+        LockService lockService,
+        JobDetailsService jobDetailsService


Why pass it to JobSweeper?
I dont see we use jobDetailsService in this file. Am I missing something?

This is preparational work, we will eventually need jobDetailsService within the JobSweeper to configure an indexToJobProviders entry with the information from the indexToJobDetails

saratvemulapalli · 2023-01-13T00:39:12Z

src/main/java/org/opensearch/jobscheduler/transport/GetJobIndexRequest.java

@@ -29,11 +29,11 @@ public class GetJobIndexRequest extends ActionRequest {

    private static String jobRunnerAction;

-    private static String extensionId;
+    private static String extensionUniqueId;

    public static final String JOB_INDEX = "job_index";


I would combine this with GetJobType and have a single request class.

This was our initial plan as well, however, we have discussed this with @owaiskazi19 and from this discussion we have decided to keep this separate. Please refer to this thread here for the rational for this design decision

saratvemulapalli · 2023-01-13T00:49:36Z

src/main/java/org/opensearch/jobscheduler/utils/JobDetailsService.java


    private static final Logger logger = LogManager.getLogger(JobDetailsService.class);
-    private static final String JOB_DETAILS_INDEX_NAME = ".opensearch-plugins-job-details";
+    public static final String JOB_DETAILS_INDEX_NAME = ".opensearch-plugins-job-details";


Nit: .opensearch-job-scheduler-job-details helps understand which plugin owns this index.

This is a good idea, I'll go ahead and change the index name

saratvemulapalli · 2023-01-13T01:00:09Z

src/main/java/org/opensearch/jobscheduler/utils/JobDetailsService.java


-    public JobDetailsService(final Client client, final ClusterService clusterService) {
+    private static final ConcurrentMap<String, JobDetails> indexToJobDetails = IndexToJobDetails.getInstance();


Trying to understand why does this have to be static?
Can we ingest this as a component to createComponent and get it from guice ?

saratvemulapalli · 2023-01-13T01:01:41Z

src/main/java/org/opensearch/jobscheduler/utils/JobDetailsService.java

+        if (jobDetails.getJobType() != null && indexToJobDetails.containsKey(extensionUniqueId)) {
+            // Update existing JobDetails entry with job type
+            JobDetails existingJobDetails = indexToJobDetails.get(extensionUniqueId);
+            existingJobDetails.setJobType(jobDetails.getJobType());


This means an extension could only register 1 job in a life time?
Wouldn't that break when an extension wants to register multiple jobs?

No, an extension will be able to register multiple jobs. The indexToJobDetails contains the initial configuration information for an extension's job to prepare the JobSweeper to begin listening to operations on the registered job index, such as the Job Index Name, Job Type, Job Runner Action name and Job Parameter Action name. This information does not change over time. Jobs are registered when extensions index a document to their corresponding job index.

…file to opensearch-job-scheduler-job-details Signed-off-by: Joshua Palis <jpalis@amazon.com>

codecov-commenter · 2023-01-18T00:12:19Z

Codecov Report

Merging #299 (0a34fd6) into main (fabb639) will decrease coverage by 8.21%.
The diff coverage is 3.14%.

@@             Coverage Diff              @@
##               main     #299      +/-   ##
============================================
- Coverage     39.38%   31.17%   -8.21%     
+ Complexity       90       66      -24     
============================================
  Files            14       12       -2     
  Lines           815      773      -42     
  Branches         84       80       -4     
============================================
- Hits            321      241      -80     
- Misses          471      513      +42     
+ Partials         23       19       -4

Impacted Files	Coverage Δ
...rg/opensearch/jobscheduler/JobSchedulerPlugin.java	`32.75% <0.00%> (-1.17%)`	⬇️
.../org/opensearch/jobscheduler/model/JobDetails.java	`0.00% <0.00%> (ø)`
...rch/jobscheduler/rest/RestGetJobDetailsAction.java	`0.00% <0.00%> (ø)`
...h/jobscheduler/transport/GetJobDetailsRequest.java	`0.00% <0.00%> (ø)`
...ensearch/jobscheduler/utils/JobDetailsService.java	`3.67% <4.76%> (-0.03%)`	⬇️
...rg/opensearch/jobscheduler/sweeper/JobSweeper.java	`39.74% <100.00%> (+0.25%)`	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

… details entry to support extensions registering multiple types of jobs, updated all integration, multinode tests now that the rest response value is the document Id Signed-off-by: Joshua Palis <jpalis@amazon.com>

saratvemulapalli

Thanks @joshpalis for the changes.

saratvemulapalli · 2023-01-19T18:11:04Z

src/test/java/org/opensearch/jobscheduler/TestHelpers.java

@@ -28,8 +28,7 @@

 public class TestHelpers {

-    public static final String GET_JOB_INDEX_BASE_DETECTORS_URI = "/_plugins/_job_scheduler/_get/_job_index";
-    public static final String GET_JOB_TYPE_BASE_DETECTORS_URI = "/_plugins/_job_scheduler/_get/_job_type";
+    public static final String GET_JOB_DETAILS_BASE_DETECTORS_URI = "/_plugins/_job_scheduler/_get/_job_details";


Why detectors ;)

Forgot to rename :)

Signed-off-by: Joshua Palis <jpalis@amazon.com>

dbwiddis

LGTM with Sarat's comments and a few suggestions and a question.

dbwiddis · 2023-01-19T18:21:26Z

src/test/java/org/opensearch/jobscheduler/multinode/GetJobDetailsMultiNodeRestIT.java

+
+        String updatedRequestBody = "{\""
+            + GetJobDetailsRequest.JOB_INDEX
+            + "\":\""


This String is screaming to be made a constant. Loudly.

Or, better yet, use String.join() with it as the delimiter.

dbwiddis · 2023-01-19T18:22:28Z

src/test/java/org/opensearch/jobscheduler/rest/RestGetJobDetailsActionIT.java

+        this.extensionUniqueId = "sample-extension";
+        this.requestBody = "{\""
+            + GetJobDetailsRequest.JOB_INDEX
+            + "\":\""


See above with regard to screaming Strings and joining.

src/test/java/org/opensearch/jobscheduler/utils/JobDetailsServiceIT.java

Signed-off-by: Joshua Palis <jpalis@amazon.com>

joshpalis · 2023-01-19T19:40:24Z

As per several offline discussions, there have been a few major changes to the job details registration workflow.

Changes to REST APIs
The SPI design allowed Job Scheduler to enforce each extending plugin to submit only one ScheduledJobProvider which includes the job index name and job type. Now that we're retrieving these values from extensions via rest apis, it is difficult to enforce extensions to invoke both the getJobIndex and getJobType sequentially, as Job Details are not indexed to the metadata index until both APIs are invoked. The discussion to keep these APIs separate can be found here, and while mirroring the plugin architecture's existing extension points may make the transition of plugins to extensions easier, separating these API calls have lead to many drawbacks and an overcomplication of the Job Details registration. As a result, we have removed the getJobType API and combined both the getJobIndex and getJobType into one request.

Changes to support extensions registering multiple job types
Previously, the metadata index in which we store job details was keyed with the extension unique ID from which the entry came from. This initial design followed the existing job registration process for plugins. In the case of extensions, there is a possibility that an extension may want to register more than one type of job, and thus in order to support an extension to register multiple times, it was necessary to forgo using the extension ID as the key. As a result, the key shall now be a randomly generated document ID which will be sent back to the caller as a rest response, in order to enable these extensions to further update their Job Details as needed.

CC : @owaiskazi19 @saratvemulapalli @vibrantvarun

…#382) * Communication mechanism for js (#289) * Job Details from Extension for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Communication Mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Communication mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Communication mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Communication mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Communication mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Commnunication Mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Commnunication Mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Commnunication Mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Commnunication Mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Commnunication Mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Commnunication Mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Communication Mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Communication Mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Communication Mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Communication Mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Communication Mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Communication Mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Communication Mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Communication Mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Communication Mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Communication Mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Communication Mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Communication Mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Communication Mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Communication Mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> Signed-off-by: Varun Jain <varunudr@amazon.com> * [Extensions] Synchronize opensearch-plugin-job-details with JobDetailsService map (#299) * Added JobDetailsService as an indexOperationListener to synchronize metadata index with internal job details m and indicesToListen set. Signed-off-by: Joshua Palis <jpalis@amazon.com> * Changes indexToJobDetails to a ConcurrentMap, adds getter method for indexToJobDetails Signed-off-by: Joshua Palis <jpalis@amazon.com> * Addressing PR comments, fixing log message Signed-off-by: Joshua Palis <jpalis@amazon.com> * Addressing PR comments Signed-off-by: Joshua Palis <jpalis@amazon.com> * Added test to updateIndexToJobDetails Signed-off-by: Joshua Palis <jpalis@amazon.com> * Addressing PR comments, changing extensionId to extensionUniqueId Signed-off-by: Joshua Palis <jpalis@amazon.com> * Addressing PR Comments : Updating Job Details index Name and mapping file to opensearch-job-scheduler-job-details Signed-off-by: Joshua Palis <jpalis@amazon.com> * Addressing PR comments, enabling extensions to submit more than 1 job details entry to support extensions registering multiple types of jobs, updated all integration, multinode tests now that the rest response value is the document Id Signed-off-by: Joshua Palis <jpalis@amazon.com> * Renaming TestHelper base URI name Signed-off-by: Joshua Palis <jpalis@amazon.com> * Addressing PR comments, made multinode test request strings constant Signed-off-by: Joshua Palis <jpalis@amazon.com> Signed-off-by: Joshua Palis <jpalis@amazon.com> * [Extensions] Create a proxy ScheduledJobRunner, ScheduledJobParser to invoke corresponding registered Job Detail actions (#306) * Draft : Added JobParameterRequest to translate ScheduledJobParser parameters nto compatible inputs for the ExtensionActionRequest. Completed initial proxy scheduled job parser implementation. Added ExtensionJobParameter class to deserialize ExtensionActionResponsebyte array into an object of type ScheduledJobParameter Signed-off-by: Joshua Palis <jpalis@amazon.com> * Added JobRunnerRequest, modified JobExecutionContext to implement writeable, created initial proxy ScheduledJobRunner, fixed failing tests Signed-off-by: Joshua Palis <jpalis@amazon.com> * Added generic ExtensionJobActionRequest that extends ExtensionActionRequest, modified JobParameter/Runner request to implement writeable, refactored proxy object creation so that the requests are added to the ExtensionJobActionRequest, which in turn upcasts the request into an ExtensionActionRequest Signed-off-by: Joshua Palis <jpalis@amazon.com> * Added byte array constructors for the JobRunner/Parameter requests, added javadocs Signed-off-by: Joshua Palis <jpalis@amazon.com> * Fixing javadocs Signed-off-by: Joshua Palis <jpalis@amazon.com> * Fixing javadocs Signed-off-by: Joshua Palis <jpalis@amazon.com> * Added placeholder string for an eventual access token to be sent to extensions to validate prior to invoking an action Signed-off-by: Joshua Palis <jpalis@amazon.com> * Added ExtensionJobActionResponse, JobParameterResponse, JobRunnerResponse to facilitate the response of extension actions Signed-off-by: Joshua Palis <jpalis@amazon.com> * Fixing javadocs Signed-off-by: Joshua Palis <jpalis@amazon.com> * Added JobRunnerResponse handling Signed-off-by: Joshua Palis <jpalis@amazon.com> * Separating updateIndexToJobProviders into separate methods Signed-off-by: Joshua Palis <jpalis@amazon.com> * Fixing javadocs Signed-off-by: Joshua Palis <jpalis@amazon.com> * SpotlessApply Signed-off-by: Joshua Palis <jpalis@amazon.com> * Addressing PR comments Signed-off-by: Joshua Palis <jpalis@amazon.com> * Added tests for serialization/deserialization of JobExecutionContext, JobDocVersion, ExtensionJobParameter, ExtensionJobActionRequest, ExtensionJobActionResponse, JobRunner/JobParameter/Request/Response Signed-off-by: Joshua Palis <jpalis@amazon.com> * Writing ExtensionActionRequest/Response to bytestream output to test deserialization Signed-off-by: Joshua Palis <jpalis@amazon.com> * Fixing imports Signed-off-by: Joshua Palis <jpalis@amazon.com> * Changing to extensionActionResponse constructor Signed-off-by: Joshua Palis <jpalis@amazon.com> * Adding tests for updateIndexToJobProviders Signed-off-by: Joshua Palis <jpalis@amazon.com> * Addressing PR comments, added getters for lock duration seconds and jitter values, added javadocs to ScheduleType enum and made this public Signed-off-by: Joshua Palis <jpalis@amazon.com> * Removing Strings dependency from lock model to fix BWC tests Signed-off-by: Joshua Palis <jpalis@amazon.com> * Fixes BWC fullRestartClusterTask, rollingUpgradeClusterTask, oldVersionCluster task. mixedClusterTask is still failing Signed-off-by: Joshua Palis <jpalis@amazon.com> * Modified createProxyScheduledJobRunner to return the document Id of the extension job parameter entry within the registered job index, modifed the ExtensionJobParameter to null check the jitter value and setting this to 0.0 if null Signed-off-by: Joshua Palis <jpalis@amazon.com> --------- Signed-off-by: Joshua Palis <jpalis@amazon.com> * [Extensions] Exposes a GetLock REST API to enable extensions to acquire a lock model for their job execution (#311) * Added RestGetLockAction API, added AcquireLockRequest, enables extensions to retrieve a lock model object to run their Job Signed-off-by: Joshua Palis <jpalis@amazon.com> * Adding Lock ID field to RestGetLockAction response Signed-off-by: Joshua Palis <jpalis@amazon.com> * Added multi node integration tests for GetLockApi Signed-off-by: Joshua Palis <jpalis@amazon.com> * Making lock service in RestGetLockAction private Signed-off-by: Joshua Palis <jpalis@amazon.com> * Added Rest integration tests for RestGetLockAction Signed-off-by: Joshua Palis <jpalis@amazon.com> * Addressing PR comments Signed-off-by: Joshua Palis <jpalis@amazon.com> * Updating get lock rest path in tests Signed-off-by: Joshua Palis <jpalis@amazon.com> * Addressing PR comments Signed-off-by: Joshua Palis <jpalis@amazon.com> --------- Signed-off-by: Joshua Palis <jpalis@amazon.com> * Communication Mechanism Work Item 6 Release lock api (#312) * Release lock API Signed-off-by: Varun Jain <varunudr@amazon.com> * Release Lock API Signed-off-by: Varun Jain <varunudr@amazon.com> * Release Lock API Signed-off-by: Varun Jain <varunudr@amazon.com> * Release Lock API Signed-off-by: Varun Jain <varunudr@amazon.com> * Reformatting Signed-off-by: Varun Jain <varunudr@amazon.com> * Reformatting Signed-off-by: Varun Jain <varunudr@amazon.com> * Reformatting Signed-off-by: Varun Jain <varunudr@amazon.com> * Addressing Dan's Comments Signed-off-by: Varun Jain <varunudr@amazon.com> * Addressing Dan's Comments Signed-off-by: Varun Jain <varunudr@amazon.com> * Addressing Sarat's Comments Signed-off-by: Varun Jain <varunudr@amazon.com> * Addressing Sarat's Comments Signed-off-by: Varun Jain <varunudr@amazon.com> * Fixing test cases Signed-off-by: Varun Jain <varunudr@amazon.com> * Fixing test cases Signed-off-by: Varun Jain <varunudr@amazon.com> * Fixing ReleaseLocktestcase Signed-off-by: Varun Jain <varunudr@amazon.com> * Fixing ReleaseLocktestcase Signed-off-by: Varun Jain <varunudr@amazon.com> --------- Signed-off-by: Varun Jain <varunudr@amazon.com> * Bumping BWC version to 2.7 and Fixing xcontent imports (#322) * [Extensions] Add all fields of LockModel to RestGetLockAction response (#342) * Modifies the RestGetLockAction response to include all fields of the lock model object, rather than the lock model object itself Signed-off-by: Joshua Palis <jpalis@amazon.com> * Removing unused fields Signed-off-by: Joshua Palis <jpalis@amazon.com> * reverting field removal Signed-off-by: Joshua Palis <jpalis@amazon.com> * Adding lockID to back to response Signed-off-by: Joshua Palis <jpalis@amazon.com> * Adding checks to multinode get lock rest integration tests to ensure that all response fields are populated Signed-off-by: Joshua Palis <jpalis@amazon.com> * fixing lock time parsing Signed-off-by: Joshua Palis <jpalis@amazon.com> * Adds a new Response class to house serde logic for RestGetLockAction response Signed-off-by: Joshua Palis <jpalis@amazon.com> * Moving parser.nextToken() within AcquireLockResponse.parse() Signed-off-by: Joshua Palis <jpalis@amazon.com> * Implementing toXContentObject for AcquireLockRequest Signed-off-by: Joshua Palis <jpalis@amazon.com> * Moving AcquireLockResponse to transpor package Signed-off-by: Joshua Palis <jpalis@amazon.com> --------- Signed-off-by: Joshua Palis <jpalis@amazon.com> * Increasing JobDetailsService request timeoufrom 10 to 15, since the initial job detail registration request will initialize the job details metadata index, which takes some time (#346) Signed-off-by: Joshua Palis <jpalis@amazon.com> * Fixes serde logic for proxy scheduled jobrunner/parser requests/responses (#349) Signed-off-by: Joshua Palis <jpalis@amazon.com> * Fixes serde logic for JobParameterRequest/JobRunnerRequest. Removes extra trimming of request bytes for the streaminput constructor for the JobParameter/RunnerRequest, as this is already trimmed by the SDK prior to forwarding the actionrequest to the transport action (#351) Signed-off-by: Joshua Palis <jpalis@amazon.com> * [Extensions] Makes JobRunner/Parameter/Request/Response classes extend from ActionRequest/Response (#352) * Replaces ExtensionActionRequest class name with the JobParameter/RunnerRequest fully qualified class names, modifes JobParameter/Runner/Request/Response classes to extend from ActionRequest/Response Signed-off-by: Joshua Palis <jpalis@amazon.com> * Removes ExtensionJobActionResponse and fixes affected test classes Signed-off-by: Joshua Palis <jpalis@amazon.com> * Fixing javadocs Signed-off-by: Joshua Palis <jpalis@amazon.com> --------- Signed-off-by: Joshua Palis <jpalis@amazon.com> * Consuming breaking changes from moving ExtensionActionRequest (#381) Signed-off-by: Sarat Vemulapalli <vemulapallisarat@gmail.com> * spotless Signed-off-by: Joshua Palis <jpalis@amazon.com> * Fixing apache imports for TestHelpers class Signed-off-by: Joshua Palis <jpalis@amazon.com> * Modofies ODFERestTestCase to work with 2.x Signed-off-by: Joshua Palis <jpalis@amazon.com> * Fixing wipeAllODFEIndices Signed-off-by: Joshua Palis <jpalis@amazon.com> --------- Signed-off-by: Varun Jain <varunudr@amazon.com> Signed-off-by: Joshua Palis <jpalis@amazon.com> Signed-off-by: Sarat Vemulapalli <vemulapallisarat@gmail.com> Co-authored-by: Varun Jain <varunudr@amazon.com> Co-authored-by: Sarat Vemulapalli <vemulapallisarat@gmail.com>

joshpalis added 2 commits January 10, 2023 20:21

Added JobDetailsService as an indexOperationListener to synchronize m…

0522cfc

…etadata index with internal job details m and indicesToListen set. Signed-off-by: Joshua Palis <jpalis@amazon.com>

Merge branch 'main' into jobrunner

9fb4d47

joshpalis requested review from dbwiddis, saratvemulapalli, vibrantvarun and a team January 10, 2023 20:28

Changes indexToJobDetails to a ConcurrentMap, adds getter method for …

cba3b5b

…indexToJobDetails Signed-off-by: Joshua Palis <jpalis@amazon.com>

vibrantvarun reviewed Jan 10, 2023

View reviewed changes

Addressing PR comments, fixing log message

1849e83

Signed-off-by: Joshua Palis <jpalis@amazon.com>

joshpalis added 2 commits January 11, 2023 00:47

Addressing PR comments

0b69d8e

Signed-off-by: Joshua Palis <jpalis@amazon.com>

Added test to updateIndexToJobDetails

5e4191b

Signed-off-by: Joshua Palis <jpalis@amazon.com>

dbwiddis reviewed Jan 11, 2023

View reviewed changes

joshpalis added 2 commits January 11, 2023 22:20

Addressing PR comments, changing extensionId to extensionUniqueId

4888a87

Signed-off-by: Joshua Palis <jpalis@amazon.com>

Merge branch 'main' into jobrunner

6dfa35b

vibrantvarun previously approved these changes Jan 12, 2023

View reviewed changes

dbwiddis previously approved these changes Jan 12, 2023

View reviewed changes

saratvemulapalli reviewed Jan 13, 2023

View reviewed changes

Addressing PR Comments : Updating Job Details index Name and mapping …

c841787

…file to opensearch-job-scheduler-job-details Signed-off-by: Joshua Palis <jpalis@amazon.com>

joshpalis dismissed stale reviews from dbwiddis and vibrantvarun via c841787 January 17, 2023 23:50

saratvemulapalli previously approved these changes Jan 19, 2023

View reviewed changes

Renaming TestHelper base URI name

4694ef8

Signed-off-by: Joshua Palis <jpalis@amazon.com>

joshpalis dismissed saratvemulapalli’s stale review via 4694ef8 January 19, 2023 18:19

dbwiddis reviewed Jan 19, 2023

View reviewed changes

saratvemulapalli previously approved these changes Jan 19, 2023

View reviewed changes

Addressing PR comments, made multinode test request strings constant

50875c4

Signed-off-by: Joshua Palis <jpalis@amazon.com>

joshpalis dismissed saratvemulapalli’s stale review via 50875c4 January 19, 2023 19:24

saratvemulapalli approved these changes Jan 19, 2023

View reviewed changes

dbwiddis approved these changes Jan 19, 2023

View reviewed changes

joshpalis merged commit 8beec22 into opensearch-project:main Jan 19, 2023

joshpalis deleted the jobrunner branch January 19, 2023 21:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Extensions] Synchronize opensearch-plugin-job-details with JobDetailsService map #299

[Extensions] Synchronize opensearch-plugin-job-details with JobDetailsService map #299

joshpalis commented Jan 10, 2023 •

edited

Loading

vibrantvarun commented Jan 10, 2023 •

edited

Loading

vibrantvarun Jan 10, 2023 •

edited

Loading

joshpalis Jan 10, 2023

joshpalis commented Jan 11, 2023 •

edited

Loading

dbwiddis left a comment

dbwiddis Jan 11, 2023

vibrantvarun commented Jan 12, 2023

dbwiddis left a comment

saratvemulapalli Jan 12, 2023

joshpalis Jan 17, 2023

saratvemulapalli Jan 12, 2023

saratvemulapalli Jan 12, 2023

joshpalis Jan 17, 2023

saratvemulapalli Jan 13, 2023

joshpalis Jan 17, 2023

saratvemulapalli Jan 13, 2023

joshpalis Jan 17, 2023

saratvemulapalli Jan 13, 2023

joshpalis Jan 17, 2023

saratvemulapalli Jan 13, 2023

saratvemulapalli Jan 13, 2023

joshpalis Jan 17, 2023

codecov-commenter commented Jan 18, 2023 •

edited

Loading

saratvemulapalli left a comment

saratvemulapalli Jan 19, 2023

joshpalis Jan 19, 2023

dbwiddis left a comment

dbwiddis Jan 19, 2023

dbwiddis Jan 19, 2023

joshpalis commented Jan 19, 2023


		public JobDetailsService(final Client client, final ClusterService clusterService) {
		private static final ConcurrentMap<String, JobDetails> indexToJobDetails = IndexToJobDetails.getInstance();

[Extensions] Synchronize opensearch-plugin-job-details with JobDetailsService map #299

[Extensions] Synchronize opensearch-plugin-job-details with JobDetailsService map #299

Conversation

joshpalis commented Jan 10, 2023 • edited Loading

Description

Issues Resolved

Check List

vibrantvarun commented Jan 10, 2023 • edited Loading

vibrantvarun Jan 10, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joshpalis commented Jan 11, 2023 • edited Loading

dbwiddis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vibrantvarun commented Jan 12, 2023

dbwiddis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-commenter commented Jan 18, 2023 • edited Loading

Codecov Report

saratvemulapalli left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dbwiddis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joshpalis commented Jan 19, 2023

joshpalis commented Jan 10, 2023 •

edited

Loading

vibrantvarun commented Jan 10, 2023 •

edited

Loading

vibrantvarun Jan 10, 2023 •

edited

Loading

joshpalis commented Jan 11, 2023 •

edited

Loading

codecov-commenter commented Jan 18, 2023 •

edited

Loading