-
Notifications
You must be signed in to change notification settings - Fork 103
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Percolate Query implementation in doc-level alerting #399
Percolate Query implementation in doc-level alerting #399
Conversation
bcc41a4
to
4e9a856
Compare
Codecov Report
@@ Coverage Diff @@
## doc-level-2.0 #399 +/- ##
================================================
Coverage ? 78.68%
Complexity ? 285
================================================
Files ? 198
Lines ? 8612
Branches ? 1148
================================================
Hits ? 6776
Misses ? 1248
Partials ? 588 Continue to review full report at Codecov.
|
fb8dee8
to
ba89f5e
Compare
9a3e538
to
826e827
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the changes. I have added few comments/clarification.
I am yet to look into the Transport Action part and will review it later today.
alerting/build.gradle
Outdated
@@ -71,7 +72,7 @@ dependencies { | |||
} | |||
|
|||
javadoc.enabled = false // turn off javadoc as it barfs on Kotlin code | |||
licenseHeaders.enabled = true | |||
licenseHeaders.enabled = false |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shouldn't we add license headers in the new files being added?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
added license to all files missing license.
import static org.opensearch.common.xcontent.ConstructingObjectParser.optionalConstructorArg; | ||
import static org.opensearch.search.SearchService.ALLOW_EXPENSIVE_QUERIES; | ||
|
||
public class PercolateQueryBuilderExt extends AbstractQueryBuilder<PercolateQueryBuilderExt> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These Ext classes seems to be the Stop Gap solutions unless we have figured out the right way to extend and use the Percolate client capabilities from the alerting plugin.
Can we add a documentation here and ToDo for cleanup?
Lets also add an issue in core repo to address the issue discovered and tag it here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
added comments to the java classes. will create an issue in core.
import static java.util.Collections.singletonList; | ||
import static java.util.Collections.singletonMap; | ||
|
||
public class PercolatorPluginExt extends Plugin implements MapperPlugin, SearchPlugin, ExtensiblePlugin { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Were we able to create the full distribution and verify the same in single and multinode setup with no issues?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes. the integration test build passes for single node & multi node with 3 nodes.
|
||
import static org.opensearch.index.query.AbstractQueryBuilder.parseInnerQueryBuilder; | ||
|
||
public class PercolatorFieldMapperExt extends ParametrizedFieldMapper { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will a change in the PercolatorFieldMapper
in core repo create the type cast exception issue with objects if this class is not kept up to date? This could be a huge maintenance overhead if not addressed shortly.
Also wondering if we also evaluated the object deserialization and serialization route for improved handling of such issues?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
not PercolatorFieldMapper
. But if api signatures from other classes in percolator-client
changes, it will need change in the copied classes too.
object deserialization and serialization route
needs to handled in core
. spi approach to make percolate plugin extensible
could also be an idea that could be pursued.
val docsToQueries = mutableMapOf<String, MutableList<String>>() | ||
val docExecutionContext = DocumentExecutionContext(queries, lastRunContext, updatedLastRunContext) | ||
val idQueryMap = mutableMapOf<String, DocLevelQuery>() | ||
queries.forEach { query -> | ||
val matchingDocIds = runForEachQuery(monitorCtx, docExecutionContext, query, index) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Lets cleanup runForEachQuery
impl.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
removed it.
logger.info("Failed to run for shard $shard. Error: ${e.message}") | ||
logger.debug("Failed to run for shard $shard", e) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
not sure with the intent of double logging here? To get the exception trace?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
removed it.
private fun getAllDocIds(hits: SearchHits): List<String> { | ||
return hits.map { hit -> hit.id } | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
replaced by getAllDocs
now?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
removed it.
): SearchHits { | ||
val percolateQueryBuilder = PercolateQueryBuilderExt("query", docs, XContentType.JSON) | ||
|
||
val searchRequest = SearchRequest(ScheduledJob.DOC_LEVEL_QUERIES_INDEX) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need a safe check if the query index is already created/exists? It could be in Runner itself. Just ensuring the exception is floated up the Runner with the right messaging
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
removed the safe check.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What happens if DOC_LEVEL_QUERIES_INDEX is not created or is deleted for some reason? I dont think we have solved this use-case yet. Not a blocker, but something for a quick follow up.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes. the current logic does not take care of high availability. This can be improved in future.
Currently, if index doesnt exist, there will be no exceptions. only no search results will be returned.
@@ -410,7 +491,47 @@ object DocumentReturningMonitorRunner : MonitorRunner { | |||
return response.hits | |||
} | |||
|
|||
private fun getMatchedQueries( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this not to get Matching Documents instead?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this method matches incoming docs with queries. named another method to getMatchingDocs
searchRequest.source(searchSourceBuilder) | ||
|
||
return if (monitorCtx.clusterService!!.state().routingTable.hasIndex(ScheduledJob.DOC_LEVEL_QUERIES_INDEX)) { | ||
val response: SearchResponse = monitorCtx.client!!.execute(SearchAction.INSTANCE, searchRequest).actionGet() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Critical - This will end up running percolate for all the incoming documents against all the queries without the filtering the monitor type against the query index. Should we have the monitor.id
field added upfront?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
passing monitor.id
as a filter also so that it filters by monitor id & then matches only those queries. https://github.com/opensearch-project/alerting/blob/6328f06451ced797f66d481f47a9fd757944f116/alerting/src/main/kotlin/org/opensearch/alerting/DocumentReturningMonitorRunner.kt#L464-#L477
826e827
to
6328f06
Compare
): SearchHits { | ||
val percolateQueryBuilder = PercolateQueryBuilderExt("query", docs, XContentType.JSON) | ||
|
||
val searchRequest = SearchRequest(ScheduledJob.DOC_LEVEL_QUERIES_INDEX) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What happens if DOC_LEVEL_QUERIES_INDEX is not created or is deleted for some reason? I dont think we have solved this use-case yet. Not a blocker, but something for a quick follow up.
@@ -409,6 +424,83 @@ class TransportIndexMonitorAction @Inject constructor( | |||
) | |||
} | |||
|
|||
@Suppress("UNCHECKED_CAST") | |||
private fun indexDocLevelMonitorQueries(monitor: Monitor, monitorId: String, refreshPolicy: RefreshPolicy) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seems to be code duplication from the monitor execution action for mapping updates. I would rather define this as a util method in one of doc level monitor related class and use at both the places if required.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
moved the duplicate code here.https://github.com/opensearch-project/alerting/blob/79a368caa73b64ab08b0cd14c4da0324ded8c1db/alerting/src/main/kotlin/org/opensearch/alerting/util/DocLevelMonitorQueries.kt#L57-#L114
this cleaned up transport logic.https://github.com/opensearch-project/alerting/blob/79a368caa73b64ab08b0cd14c4da0324ded8c1db/alerting/src/main/kotlin/org/opensearch/alerting/transport/TransportIndexMonitorAction.kt#L425-#L474
@@ -74,6 +91,65 @@ class TransportExecuteMonitorAction @Inject constructor( | |||
} | |||
} | |||
} | |||
val indexDocLevelMonitorQueries = fun(monitor: Monitor, monitorId: String) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we really need to update the mapping each time during the monitor run execution? Shouldn't the Index Monitor path take care of any update in the mapping?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
there are 2 types of execute action. one which takes already created monitor_id
as input. the other one directly takes entire monitor payload as input.
update mapping is needed in 2nd case.
https://github.com/opensearch-project/alerting/blob/79a368caa73b64ab08b0cd14c4da0324ded8c1db/alerting/src/main/kotlin/org/opensearch/alerting/transport/TransportExecuteMonitorAction.kt#L89-#L180
79a368c
to
a55fe04
Compare
@@ -0,0 +1,13 @@ | |||
{ | |||
"_meta": { | |||
"schema_version": 5 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
change this to 1
a55fe04
to
41d962b
Compare
Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>
41d962b
to
0bab665
Compare
* Rebase to push doc level changes on latest main changes (#391) * Document level alerting dev (#272) Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * Add last run context to Monitor data model Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * add Update Monitor function Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * fix integ test Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * Implemented draft of Finding data model, a new Input type, and some basic unit tests. (#260) * Implemented draft of Finding data model, and some basic unit tests for it. Signed-off-by: AWSHurneyt <79280347+AWSHurneyt@users.noreply.github.com> * POC for doc-level-alerting (#277) Signed-off-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com> * Add connection to triggers for doc level alerting (#316) Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * CRUD APIs integration Tests and validation"conflict resolved" (#362) Signed-off-by: charliezhangaws <zhanncha@amazon.com> * Segregate monitor runner logic for separation of concerns (#363) * Refactor monitor runner logic for separation of concerns and better testability. Signed-off-by: Saurabh Singh <getsaurabh02@gmail.com> * Add action and alert flow and findings schema and additional fixes (#381) Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * Finding Search API (#385) * Findings search API based on Annie's work Signed-off-by: Annie Lee <leeyun@amazon.com> * Fix Search API and add IT tests Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> Co-authored-by: Annie Lee <leeyun@amazon.com> * Fix integ tests and minor issues from doc level changes Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> Co-authored-by: Annie Lee <71157062+leeyun-amzn@users.noreply.github.com> Co-authored-by: Daniel Doubrovkine (dB.) <dblock@dblock.org> Co-authored-by: AWSHurneyt <79280347+AWSHurneyt@users.noreply.github.com> Co-authored-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com> Co-authored-by: charliezhangaws <zhanncha@amazon.com> Co-authored-by: Saurabh Singh <getsaurabh02@gmail.com> Co-authored-by: Annie Lee <leeyun@amazon.com> * Add Trigger condition resolver which parses and evaluates the Trigger expression. (#405) Signed-off-by: Saurabh Singh <sisurab@amazon.com> * percolate query implementation in doc-level alerting (#399) Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> * Finding Index rollover (#408) * Finding Index rollover Signed-off-by: jiahe zhang <zhanncha@amazon.com> * Apply fixes to make rollover work Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> Co-authored-by: jiahe zhang <zhanncha@amazon.com> Co-authored-by: Annie Lee <71157062+leeyun-amzn@users.noreply.github.com> Co-authored-by: Daniel Doubrovkine (dB.) <dblock@dblock.org> Co-authored-by: AWSHurneyt <79280347+AWSHurneyt@users.noreply.github.com> Co-authored-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com> Co-authored-by: charliezhangaws <zhanncha@amazon.com> Co-authored-by: Saurabh Singh <getsaurabh02@gmail.com> Co-authored-by: Annie Lee <leeyun@amazon.com> Co-authored-by: Saurabh Singh <sisurab@amazon.com> Co-authored-by: Subhobrata Dey <sbcd90@gmail.com>
* Rebase to push doc level changes on latest main changes (opensearch-project#391) * Document level alerting dev (opensearch-project#272) Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * Add last run context to Monitor data model Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * add Update Monitor function Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * fix integ test Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * Implemented draft of Finding data model, a new Input type, and some basic unit tests. (opensearch-project#260) * Implemented draft of Finding data model, and some basic unit tests for it. Signed-off-by: AWSHurneyt <79280347+AWSHurneyt@users.noreply.github.com> * POC for doc-level-alerting (opensearch-project#277) Signed-off-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com> * Add connection to triggers for doc level alerting (opensearch-project#316) Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * CRUD APIs integration Tests and validation"conflict resolved" (opensearch-project#362) Signed-off-by: charliezhangaws <zhanncha@amazon.com> * Segregate monitor runner logic for separation of concerns (opensearch-project#363) * Refactor monitor runner logic for separation of concerns and better testability. Signed-off-by: Saurabh Singh <getsaurabh02@gmail.com> * Add action and alert flow and findings schema and additional fixes (opensearch-project#381) Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * Finding Search API (opensearch-project#385) * Findings search API based on Annie's work Signed-off-by: Annie Lee <leeyun@amazon.com> * Fix Search API and add IT tests Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> Co-authored-by: Annie Lee <leeyun@amazon.com> * Fix integ tests and minor issues from doc level changes Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> Co-authored-by: Annie Lee <71157062+leeyun-amzn@users.noreply.github.com> Co-authored-by: Daniel Doubrovkine (dB.) <dblock@dblock.org> Co-authored-by: AWSHurneyt <79280347+AWSHurneyt@users.noreply.github.com> Co-authored-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com> Co-authored-by: charliezhangaws <zhanncha@amazon.com> Co-authored-by: Saurabh Singh <getsaurabh02@gmail.com> Co-authored-by: Annie Lee <leeyun@amazon.com> * Add Trigger condition resolver which parses and evaluates the Trigger expression. (opensearch-project#405) Signed-off-by: Saurabh Singh <sisurab@amazon.com> * percolate query implementation in doc-level alerting (opensearch-project#399) Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> * Finding Index rollover (opensearch-project#408) * Finding Index rollover Signed-off-by: jiahe zhang <zhanncha@amazon.com> * Apply fixes to make rollover work Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> Co-authored-by: jiahe zhang <zhanncha@amazon.com> Co-authored-by: Annie Lee <71157062+leeyun-amzn@users.noreply.github.com> Co-authored-by: Daniel Doubrovkine (dB.) <dblock@dblock.org> Co-authored-by: AWSHurneyt <79280347+AWSHurneyt@users.noreply.github.com> Co-authored-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com> Co-authored-by: charliezhangaws <zhanncha@amazon.com> Co-authored-by: Saurabh Singh <getsaurabh02@gmail.com> Co-authored-by: Annie Lee <leeyun@amazon.com> Co-authored-by: Saurabh Singh <sisurab@amazon.com> Co-authored-by: Subhobrata Dey <sbcd90@gmail.com> Signed-off-by: AWSHurneyt <hurneyt@amazon.com>
* fix security test workflow (#407) Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Integrate Document Level Alerting changes (#410) * Rebase to push doc level changes on latest main changes (#391) * Document level alerting dev (#272) Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * Add last run context to Monitor data model Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * add Update Monitor function Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * fix integ test Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * Implemented draft of Finding data model, a new Input type, and some basic unit tests. (#260) * Implemented draft of Finding data model, and some basic unit tests for it. Signed-off-by: AWSHurneyt <79280347+AWSHurneyt@users.noreply.github.com> * POC for doc-level-alerting (#277) Signed-off-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com> * Add connection to triggers for doc level alerting (#316) Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * CRUD APIs integration Tests and validation"conflict resolved" (#362) Signed-off-by: charliezhangaws <zhanncha@amazon.com> * Segregate monitor runner logic for separation of concerns (#363) * Refactor monitor runner logic for separation of concerns and better testability. Signed-off-by: Saurabh Singh <getsaurabh02@gmail.com> * Add action and alert flow and findings schema and additional fixes (#381) Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * Finding Search API (#385) * Findings search API based on Annie's work Signed-off-by: Annie Lee <leeyun@amazon.com> * Fix Search API and add IT tests Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> Co-authored-by: Annie Lee <leeyun@amazon.com> * Fix integ tests and minor issues from doc level changes Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> Co-authored-by: Annie Lee <71157062+leeyun-amzn@users.noreply.github.com> Co-authored-by: Daniel Doubrovkine (dB.) <dblock@dblock.org> Co-authored-by: AWSHurneyt <79280347+AWSHurneyt@users.noreply.github.com> Co-authored-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com> Co-authored-by: charliezhangaws <zhanncha@amazon.com> Co-authored-by: Saurabh Singh <getsaurabh02@gmail.com> Co-authored-by: Annie Lee <leeyun@amazon.com> * Add Trigger condition resolver which parses and evaluates the Trigger expression. (#405) Signed-off-by: Saurabh Singh <sisurab@amazon.com> * percolate query implementation in doc-level alerting (#399) Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> * Finding Index rollover (#408) * Finding Index rollover Signed-off-by: jiahe zhang <zhanncha@amazon.com> * Apply fixes to make rollover work Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> Co-authored-by: jiahe zhang <zhanncha@amazon.com> Co-authored-by: Annie Lee <71157062+leeyun-amzn@users.noreply.github.com> Co-authored-by: Daniel Doubrovkine (dB.) <dblock@dblock.org> Co-authored-by: AWSHurneyt <79280347+AWSHurneyt@users.noreply.github.com> Co-authored-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com> Co-authored-by: charliezhangaws <zhanncha@amazon.com> Co-authored-by: Saurabh Singh <getsaurabh02@gmail.com> Co-authored-by: Annie Lee <leeyun@amazon.com> Co-authored-by: Saurabh Singh <sisurab@amazon.com> Co-authored-by: Subhobrata Dey <sbcd90@gmail.com> Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Remove write Destination APIs (#412) * Remove write Destination API REST handlers Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com> * Remove write Destination transport actions Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com> * Remove write Destination action, request and response classes Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com> * Create Alerting config index if it doesn't exist before legacy Destination indexing Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com> * Remove write Destination related security tests Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com> * Remove unused access roles and imports Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com> Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Deprecate the Master nomenclature in 2.0 (#415) Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Remove Alerting's notification subproject (#413) * Remove notification subproject Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com> * Remove publishing to maven in build script Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com> Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Skipping destination migration if alerting index is not initialized (#417) Signed-off-by: Ravi 6005951+thalurur@users.noreply.github.com Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Alias support for Document Level Monitors (#416) * Implemented support for defining doc level monitors using aliases. Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Fix integ tests and cleaup alias logic Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> Co-authored-by: AWSHurneyt <hurneyt@amazon.com> Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Fixed a flaky test condition. (#375) Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Replace checked-in ZIP for bwc tests with a dynamic dependency (#411) * Replace checked-in ZIP with a dynamic dependency Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> * Replace checked-in ZIP with a dynamic dependency Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Update integTest gradle scripts to run via remote cluster independently (#418) Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Removed tests that were duplicated while resolving merge conflicts. Signed-off-by: AWSHurneyt <hurneyt@amazon.com> Co-authored-by: Subhobrata Dey <sbcd90@gmail.com> Co-authored-by: Ashish Agrawal <ashisagr@amazon.com> Co-authored-by: Annie Lee <71157062+leeyun-amzn@users.noreply.github.com> Co-authored-by: Daniel Doubrovkine (dB.) <dblock@dblock.org> Co-authored-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com> Co-authored-by: charliezhangaws <zhanncha@amazon.com> Co-authored-by: Saurabh Singh <getsaurabh02@gmail.com> Co-authored-by: Annie Lee <leeyun@amazon.com> Co-authored-by: Saurabh Singh <sisurab@amazon.com> Co-authored-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com> Co-authored-by: Ravi <6005951+thalurur@users.noreply.github.com>
* fix security test workflow (opensearch-project#407) Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Integrate Document Level Alerting changes (opensearch-project#410) * Rebase to push doc level changes on latest main changes (opensearch-project#391) * Document level alerting dev (opensearch-project#272) Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * Add last run context to Monitor data model Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * add Update Monitor function Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * fix integ test Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * Implemented draft of Finding data model, a new Input type, and some basic unit tests. (opensearch-project#260) * Implemented draft of Finding data model, and some basic unit tests for it. Signed-off-by: AWSHurneyt <79280347+AWSHurneyt@users.noreply.github.com> * POC for doc-level-alerting (opensearch-project#277) Signed-off-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com> * Add connection to triggers for doc level alerting (opensearch-project#316) Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * CRUD APIs integration Tests and validation"conflict resolved" (opensearch-project#362) Signed-off-by: charliezhangaws <zhanncha@amazon.com> * Segregate monitor runner logic for separation of concerns (opensearch-project#363) * Refactor monitor runner logic for separation of concerns and better testability. Signed-off-by: Saurabh Singh <getsaurabh02@gmail.com> * Add action and alert flow and findings schema and additional fixes (opensearch-project#381) Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * Finding Search API (opensearch-project#385) * Findings search API based on Annie's work Signed-off-by: Annie Lee <leeyun@amazon.com> * Fix Search API and add IT tests Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> Co-authored-by: Annie Lee <leeyun@amazon.com> * Fix integ tests and minor issues from doc level changes Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> Co-authored-by: Annie Lee <71157062+leeyun-amzn@users.noreply.github.com> Co-authored-by: Daniel Doubrovkine (dB.) <dblock@dblock.org> Co-authored-by: AWSHurneyt <79280347+AWSHurneyt@users.noreply.github.com> Co-authored-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com> Co-authored-by: charliezhangaws <zhanncha@amazon.com> Co-authored-by: Saurabh Singh <getsaurabh02@gmail.com> Co-authored-by: Annie Lee <leeyun@amazon.com> * Add Trigger condition resolver which parses and evaluates the Trigger expression. (opensearch-project#405) Signed-off-by: Saurabh Singh <sisurab@amazon.com> * percolate query implementation in doc-level alerting (opensearch-project#399) Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> * Finding Index rollover (opensearch-project#408) * Finding Index rollover Signed-off-by: jiahe zhang <zhanncha@amazon.com> * Apply fixes to make rollover work Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> Co-authored-by: jiahe zhang <zhanncha@amazon.com> Co-authored-by: Annie Lee <71157062+leeyun-amzn@users.noreply.github.com> Co-authored-by: Daniel Doubrovkine (dB.) <dblock@dblock.org> Co-authored-by: AWSHurneyt <79280347+AWSHurneyt@users.noreply.github.com> Co-authored-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com> Co-authored-by: charliezhangaws <zhanncha@amazon.com> Co-authored-by: Saurabh Singh <getsaurabh02@gmail.com> Co-authored-by: Annie Lee <leeyun@amazon.com> Co-authored-by: Saurabh Singh <sisurab@amazon.com> Co-authored-by: Subhobrata Dey <sbcd90@gmail.com> Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Remove write Destination APIs (opensearch-project#412) * Remove write Destination API REST handlers Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com> * Remove write Destination transport actions Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com> * Remove write Destination action, request and response classes Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com> * Create Alerting config index if it doesn't exist before legacy Destination indexing Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com> * Remove write Destination related security tests Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com> * Remove unused access roles and imports Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com> Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Deprecate the Master nomenclature in 2.0 (opensearch-project#415) Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Remove Alerting's notification subproject (opensearch-project#413) * Remove notification subproject Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com> * Remove publishing to maven in build script Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com> Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Skipping destination migration if alerting index is not initialized (opensearch-project#417) Signed-off-by: Ravi 6005951+thalurur@users.noreply.github.com Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Alias support for Document Level Monitors (opensearch-project#416) * Implemented support for defining doc level monitors using aliases. Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Fix integ tests and cleaup alias logic Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> Co-authored-by: AWSHurneyt <hurneyt@amazon.com> Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Fixed a flaky test condition. (opensearch-project#375) Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Replace checked-in ZIP for bwc tests with a dynamic dependency (opensearch-project#411) * Replace checked-in ZIP with a dynamic dependency Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> * Replace checked-in ZIP with a dynamic dependency Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Update integTest gradle scripts to run via remote cluster independently (opensearch-project#418) Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Removed tests that were duplicated while resolving merge conflicts. Signed-off-by: AWSHurneyt <hurneyt@amazon.com> Co-authored-by: Subhobrata Dey <sbcd90@gmail.com> Co-authored-by: Ashish Agrawal <ashisagr@amazon.com> Co-authored-by: Annie Lee <71157062+leeyun-amzn@users.noreply.github.com> Co-authored-by: Daniel Doubrovkine (dB.) <dblock@dblock.org> Co-authored-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com> Co-authored-by: charliezhangaws <zhanncha@amazon.com> Co-authored-by: Saurabh Singh <getsaurabh02@gmail.com> Co-authored-by: Annie Lee <leeyun@amazon.com> Co-authored-by: Saurabh Singh <sisurab@amazon.com> Co-authored-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com> Co-authored-by: Ravi <6005951+thalurur@users.noreply.github.com> Signed-off-by: Angie Zhang <langelzh@amazon.com>
* Rebase to push doc level changes on latest main changes (opensearch-project#391) * Document level alerting dev (opensearch-project#272) Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * Add last run context to Monitor data model Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * add Update Monitor function Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * fix integ test Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * Implemented draft of Finding data model, a new Input type, and some basic unit tests. (opensearch-project#260) * Implemented draft of Finding data model, and some basic unit tests for it. Signed-off-by: AWSHurneyt <79280347+AWSHurneyt@users.noreply.github.com> * POC for doc-level-alerting (opensearch-project#277) Signed-off-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com> * Add connection to triggers for doc level alerting (opensearch-project#316) Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * CRUD APIs integration Tests and validation"conflict resolved" (opensearch-project#362) Signed-off-by: charliezhangaws <zhanncha@amazon.com> * Segregate monitor runner logic for separation of concerns (opensearch-project#363) * Refactor monitor runner logic for separation of concerns and better testability. Signed-off-by: Saurabh Singh <getsaurabh02@gmail.com> * Add action and alert flow and findings schema and additional fixes (opensearch-project#381) Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * Finding Search API (opensearch-project#385) * Findings search API based on Annie's work Signed-off-by: Annie Lee <leeyun@amazon.com> * Fix Search API and add IT tests Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> Co-authored-by: Annie Lee <leeyun@amazon.com> * Fix integ tests and minor issues from doc level changes Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> Co-authored-by: Annie Lee <71157062+leeyun-amzn@users.noreply.github.com> Co-authored-by: Daniel Doubrovkine (dB.) <dblock@dblock.org> Co-authored-by: AWSHurneyt <79280347+AWSHurneyt@users.noreply.github.com> Co-authored-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com> Co-authored-by: charliezhangaws <zhanncha@amazon.com> Co-authored-by: Saurabh Singh <getsaurabh02@gmail.com> Co-authored-by: Annie Lee <leeyun@amazon.com> * Add Trigger condition resolver which parses and evaluates the Trigger expression. (opensearch-project#405) Signed-off-by: Saurabh Singh <sisurab@amazon.com> * percolate query implementation in doc-level alerting (opensearch-project#399) Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> * Finding Index rollover (opensearch-project#408) * Finding Index rollover Signed-off-by: jiahe zhang <zhanncha@amazon.com> * Apply fixes to make rollover work Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> Co-authored-by: jiahe zhang <zhanncha@amazon.com> Co-authored-by: Annie Lee <71157062+leeyun-amzn@users.noreply.github.com> Co-authored-by: Daniel Doubrovkine (dB.) <dblock@dblock.org> Co-authored-by: AWSHurneyt <79280347+AWSHurneyt@users.noreply.github.com> Co-authored-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com> Co-authored-by: charliezhangaws <zhanncha@amazon.com> Co-authored-by: Saurabh Singh <getsaurabh02@gmail.com> Co-authored-by: Annie Lee <leeyun@amazon.com> Co-authored-by: Saurabh Singh <sisurab@amazon.com> Co-authored-by: Subhobrata Dey <sbcd90@gmail.com> Signed-off-by: Angie Zhang <langelzh@amazon.com>
* fix security test workflow (opensearch-project#407) Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Integrate Document Level Alerting changes (opensearch-project#410) * Rebase to push doc level changes on latest main changes (opensearch-project#391) * Document level alerting dev (opensearch-project#272) Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * Add last run context to Monitor data model Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * add Update Monitor function Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * fix integ test Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * Implemented draft of Finding data model, a new Input type, and some basic unit tests. (opensearch-project#260) * Implemented draft of Finding data model, and some basic unit tests for it. Signed-off-by: AWSHurneyt <79280347+AWSHurneyt@users.noreply.github.com> * POC for doc-level-alerting (opensearch-project#277) Signed-off-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com> * Add connection to triggers for doc level alerting (opensearch-project#316) Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * CRUD APIs integration Tests and validation"conflict resolved" (opensearch-project#362) Signed-off-by: charliezhangaws <zhanncha@amazon.com> * Segregate monitor runner logic for separation of concerns (opensearch-project#363) * Refactor monitor runner logic for separation of concerns and better testability. Signed-off-by: Saurabh Singh <getsaurabh02@gmail.com> * Add action and alert flow and findings schema and additional fixes (opensearch-project#381) Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * Finding Search API (opensearch-project#385) * Findings search API based on Annie's work Signed-off-by: Annie Lee <leeyun@amazon.com> * Fix Search API and add IT tests Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> Co-authored-by: Annie Lee <leeyun@amazon.com> * Fix integ tests and minor issues from doc level changes Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> Co-authored-by: Annie Lee <71157062+leeyun-amzn@users.noreply.github.com> Co-authored-by: Daniel Doubrovkine (dB.) <dblock@dblock.org> Co-authored-by: AWSHurneyt <79280347+AWSHurneyt@users.noreply.github.com> Co-authored-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com> Co-authored-by: charliezhangaws <zhanncha@amazon.com> Co-authored-by: Saurabh Singh <getsaurabh02@gmail.com> Co-authored-by: Annie Lee <leeyun@amazon.com> * Add Trigger condition resolver which parses and evaluates the Trigger expression. (opensearch-project#405) Signed-off-by: Saurabh Singh <sisurab@amazon.com> * percolate query implementation in doc-level alerting (opensearch-project#399) Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> * Finding Index rollover (opensearch-project#408) * Finding Index rollover Signed-off-by: jiahe zhang <zhanncha@amazon.com> * Apply fixes to make rollover work Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> Co-authored-by: jiahe zhang <zhanncha@amazon.com> Co-authored-by: Annie Lee <71157062+leeyun-amzn@users.noreply.github.com> Co-authored-by: Daniel Doubrovkine (dB.) <dblock@dblock.org> Co-authored-by: AWSHurneyt <79280347+AWSHurneyt@users.noreply.github.com> Co-authored-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com> Co-authored-by: charliezhangaws <zhanncha@amazon.com> Co-authored-by: Saurabh Singh <getsaurabh02@gmail.com> Co-authored-by: Annie Lee <leeyun@amazon.com> Co-authored-by: Saurabh Singh <sisurab@amazon.com> Co-authored-by: Subhobrata Dey <sbcd90@gmail.com> Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Remove write Destination APIs (opensearch-project#412) * Remove write Destination API REST handlers Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com> * Remove write Destination transport actions Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com> * Remove write Destination action, request and response classes Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com> * Create Alerting config index if it doesn't exist before legacy Destination indexing Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com> * Remove write Destination related security tests Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com> * Remove unused access roles and imports Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com> Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Deprecate the Master nomenclature in 2.0 (opensearch-project#415) Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Remove Alerting's notification subproject (opensearch-project#413) * Remove notification subproject Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com> * Remove publishing to maven in build script Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com> Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Skipping destination migration if alerting index is not initialized (opensearch-project#417) Signed-off-by: Ravi 6005951+thalurur@users.noreply.github.com Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Alias support for Document Level Monitors (opensearch-project#416) * Implemented support for defining doc level monitors using aliases. Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Fix integ tests and cleaup alias logic Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> Co-authored-by: AWSHurneyt <hurneyt@amazon.com> Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Fixed a flaky test condition. (opensearch-project#375) Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Replace checked-in ZIP for bwc tests with a dynamic dependency (opensearch-project#411) * Replace checked-in ZIP with a dynamic dependency Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> * Replace checked-in ZIP with a dynamic dependency Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Update integTest gradle scripts to run via remote cluster independently (opensearch-project#418) Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Removed tests that were duplicated while resolving merge conflicts. Signed-off-by: AWSHurneyt <hurneyt@amazon.com> Co-authored-by: Subhobrata Dey <sbcd90@gmail.com> Co-authored-by: Ashish Agrawal <ashisagr@amazon.com> Co-authored-by: Annie Lee <71157062+leeyun-amzn@users.noreply.github.com> Co-authored-by: Daniel Doubrovkine (dB.) <dblock@dblock.org> Co-authored-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com> Co-authored-by: charliezhangaws <zhanncha@amazon.com> Co-authored-by: Saurabh Singh <getsaurabh02@gmail.com> Co-authored-by: Annie Lee <leeyun@amazon.com> Co-authored-by: Saurabh Singh <sisurab@amazon.com> Co-authored-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com> Co-authored-by: Ravi <6005951+thalurur@users.noreply.github.com> Signed-off-by: Angie Zhang <langelzh@amazon.com>
percolate query implementation in doc-level alerting
Issue #, if available:
Percolate Query implementation in Document Level Alerting
Description of changes:
This PR provides Percolate Query based implementation in Document Level Alerting. It implements the following design with its limitations.
CheckList:
[X] Commits are signed per the DCO using --signoff
[X] Integration tests work for doc-level alerting
[ ] Performance test results - pending
[ ] Additional tests
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.