Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[META] Implement a Communication Mechanism Between Job Scheduler as a Plugin and Anomaly Detection as an Extension #5585

Closed
6 tasks done
joshpalis opened this issue Nov 24, 2022 · 0 comments
Assignees
Labels
enhancement Enhancement or improvement to existing feature or request

Comments

@joshpalis
Copy link
Member

joshpalis commented Nov 24, 2022

Problem

Anomaly Detection currently takes Job Scheduler as a dependency in order to schedule Anomaly Detectors to run for a specified time interval. Job Scheduler has an SPI (Service Provider Interface) design and provides the JobSchedulerExtension interface which Anomaly Detection implements for its own Job Runner. Job Scheduler also implements the Extensible Plugin Interface which provides the capability to class load Anomaly Detection's implementations for the Job Scheduler Extension. This is the currently plugin architecture that enables Anomaly Detection to use Job Scheduler to schedule its own jobs, however in the case of Anomaly Detection as an extension running in a separate JVM process, the SPI design is not compatible.

Objective

In order to provide Anomaly Detection as an extension with the capability to schedule its own jobs, we must design and develop a communication mechanism between Anomaly Detection Extension and Job Scheduler as a plugin installed within OpenSearch. This communication mechanism should be integrated alongside the current plugin architecture in order to support both plugins and extensions.

AD Features Dependent on Job Scheduler Integration

In order to achieve support for Anomaly Detection Extension to schedule jobs, the communication mechanism must be able to facilitate the workflow triggered by the following APIs.

Start Detector Workflow

The Start Detector API triggers the process in which Anomaly Detector Jobs are configured and scheduled, therefore it is important to first understand the interactions of Job Scheduler and Anomaly Detection throughout this process within the current plugin architecture in order to determine how to build around this. The following workflow diagram describes how Job Scheduler class loads AD's JobSchedulerExtension implementations and also traces the events that invoke this job scheduling and execution.

Note : For Anomaly Detector jobs, Job Scheduler is used only to schedule and execute real-time anomaly detectors, therefore we have omitted exploration of the historical analysis workflow.

Design for Communication Mechanism between Plugins and Extensions

Issues will be created and linked here

Note : The work items above correspond with the purple boxes found on the start detector workflow diagram, which indicate when these various request/responses should executed

StartDetector (3)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Enhancement or improvement to existing feature or request
Projects
None yet
Development

No branches or pull requests

3 participants