Backporting changes to 2.0 (opensearch-project#420)

* fix security test workflow (opensearch-project#407) Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Integrate Document Level Alerting changes (opensearch-project#410) * Rebase to push doc level changes on latest main changes (opensearch-project#391) * Document level alerting dev (opensearch-project#272) Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * Add last run context to Monitor data model Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * add Update Monitor function Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * fix integ test Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * Implemented draft of Finding data model, a new Input type, and some basic unit tests. (opensearch-project#260) * Implemented draft of Finding data model, and some basic unit tests for it. Signed-off-by: AWSHurneyt <79280347+AWSHurneyt@users.noreply.github.com> * POC for doc-level-alerting (opensearch-project#277) Signed-off-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com> * Add connection to triggers for doc level alerting (opensearch-project#316) Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * CRUD APIs integration Tests and validation"conflict resolved" (opensearch-project#362) Signed-off-by: charliezhangaws <zhanncha@amazon.com> * Segregate monitor runner logic for separation of concerns (opensearch-project#363) * Refactor monitor runner logic for separation of concerns and better testability. Signed-off-by: Saurabh Singh <getsaurabh02@gmail.com> * Add action and alert flow and findings schema and additional fixes (opensearch-project#381) Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * Finding Search API (opensearch-project#385) * Findings search API based on Annie's work Signed-off-by: Annie Lee <leeyun@amazon.com> * Fix Search API and add IT tests Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> Co-authored-by: Annie Lee <leeyun@amazon.com> * Fix integ tests and minor issues from doc level changes Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> Co-authored-by: Annie Lee <71157062+leeyun-amzn@users.noreply.github.com> Co-authored-by: Daniel Doubrovkine (dB.) <dblock@dblock.org> Co-authored-by: AWSHurneyt <79280347+AWSHurneyt@users.noreply.github.com> Co-authored-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com> Co-authored-by: charliezhangaws <zhanncha@amazon.com> Co-authored-by: Saurabh Singh <getsaurabh02@gmail.com> Co-authored-by: Annie Lee <leeyun@amazon.com> * Add Trigger condition resolver which parses and evaluates the Trigger expression. (opensearch-project#405) Signed-off-by: Saurabh Singh <sisurab@amazon.com> * percolate query implementation in doc-level alerting (opensearch-project#399) Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> * Finding Index rollover (opensearch-project#408) * Finding Index rollover Signed-off-by: jiahe zhang <zhanncha@amazon.com> * Apply fixes to make rollover work Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> Co-authored-by: jiahe zhang <zhanncha@amazon.com> Co-authored-by: Annie Lee <71157062+leeyun-amzn@users.noreply.github.com> Co-authored-by: Daniel Doubrovkine (dB.) <dblock@dblock.org> Co-authored-by: AWSHurneyt <79280347+AWSHurneyt@users.noreply.github.com> Co-authored-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com> Co-authored-by: charliezhangaws <zhanncha@amazon.com> Co-authored-by: Saurabh Singh <getsaurabh02@gmail.com> Co-authored-by: Annie Lee <leeyun@amazon.com> Co-authored-by: Saurabh Singh <sisurab@amazon.com> Co-authored-by: Subhobrata Dey <sbcd90@gmail.com> Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Remove write Destination APIs (opensearch-project#412) * Remove write Destination API REST handlers Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com> * Remove write Destination transport actions Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com> * Remove write Destination action, request and response classes Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com> * Create Alerting config index if it doesn't exist before legacy Destination indexing Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com> * Remove write Destination related security tests Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com> * Remove unused access roles and imports Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com> Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Deprecate the Master nomenclature in 2.0 (opensearch-project#415) Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Remove Alerting's notification subproject (opensearch-project#413) * Remove notification subproject Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com> * Remove publishing to maven in build script Signed-off-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com> Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Skipping destination migration if alerting index is not initialized (opensearch-project#417) Signed-off-by: Ravi 6005951+thalurur@users.noreply.github.com Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Alias support for Document Level Monitors (opensearch-project#416) * Implemented support for defining doc level monitors using aliases. Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Fix integ tests and cleaup alias logic Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> Co-authored-by: AWSHurneyt <hurneyt@amazon.com> Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Fixed a flaky test condition. (opensearch-project#375) Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Replace checked-in ZIP for bwc tests with a dynamic dependency (opensearch-project#411) * Replace checked-in ZIP with a dynamic dependency Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> * Replace checked-in ZIP with a dynamic dependency Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Update integTest gradle scripts to run via remote cluster independently (opensearch-project#418) Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Removed tests that were duplicated while resolving merge conflicts. Signed-off-by: AWSHurneyt <hurneyt@amazon.com> Co-authored-by: Subhobrata Dey <sbcd90@gmail.com> Co-authored-by: Ashish Agrawal <ashisagr@amazon.com> Co-authored-by: Annie Lee <71157062+leeyun-amzn@users.noreply.github.com> Co-authored-by: Daniel Doubrovkine (dB.) <dblock@dblock.org> Co-authored-by: Sriram <59816283+skkosuri-amzn@users.noreply.github.com> Co-authored-by: charliezhangaws <zhanncha@amazon.com> Co-authored-by: Saurabh Singh <getsaurabh02@gmail.com> Co-authored-by: Annie Lee <leeyun@amazon.com> Co-authored-by: Saurabh Singh <sisurab@amazon.com> Co-authored-by: Mohammad Qureshi <47198598+qreshi@users.noreply.github.com> Co-authored-by: Ravi <6005951+thalurur@users.noreply.github.com> Signed-off-by: Angie Zhang <langelzh@amazon.com>
Angie-Zhang · Jun 28, 2022 · 01e12e1 · 01e12e1
1 parent 214c9db
commit 01e12e1
Show file tree

Hide file tree

Showing 24 changed files with 697 additions and 179 deletions.
diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md
@@ -63,10 +63,12 @@ Currently we just put RCF jar in lib as dependency. Plan to publish to Maven and
 6. `./gradlew :alerting:integTest -Dtests.class="*MonitorRunnerIT"` runs a single integ test class
 7. `./gradlew :alerting:integTest -Dtests.method="test execute monitor with dryrun"` runs a single integ test method
  (remember to quote the test method name if it contains spaces).
-8. `./gradlew alertingBwcCluster#mixedClusterTask` launches a cluster with three nodes of bwc version of OpenSearch with alerting and tests backwards compatibility by upgrading one of the nodes with the current version of OpenSearch with alerting, creating a mixed cluster.
-9. `./gradlew alertingBwcCluster#rollingUpgradeClusterTask` launches a cluster with three nodes of bwc version of OpenSearch with alerting and tests backwards compatibility by performing rolling upgrade of each node with the current version of OpenSearch with alerting.
-10. `./gradlew alertingBwcCluster#fullRestartClusterTask` launches a cluster with three nodes of bwc version of OpenSearch with alerting and tests backwards compatibility by performing a full restart on the cluster upgrading all the nodes with the current version of OpenSearch with alerting.
-11. `./gradlew bwcTestSuite` runs all the above bwc tests combined.
+8. `./gradlew :alerting:integTestRemote -Dtests.rest.cluster=localhost:9200 -Dtests.cluster=localhost:9200 -Dtests.clustername=docker-cluster` launches integration tests against a local cluster
+9. `./gradlew :alerting:integTestRemote -Dtests.rest.cluster=localhost:9200 -Dtests.cluster=localhost:9200 -Dtests.clustername=docker-cluster -Dsecurity=true -Dhttps=true -Duser=admin -Dpassword=admin` launches integration tests against a local cluster and run tests with security
+10. `./gradlew alertingBwcCluster#mixedClusterTask` launches a cluster with three nodes of bwc version of OpenSearch with alerting and tests backwards compatibility by upgrading one of the nodes with the current version of OpenSearch with alerting, creating a mixed cluster.
+11. `./gradlew alertingBwcCluster#rollingUpgradeClusterTask` launches a cluster with three nodes of bwc version of OpenSearch with alerting and tests backwards compatibility by performing rolling upgrade of each node with the current version of OpenSearch with alerting.
+12. `./gradlew alertingBwcCluster#fullRestartClusterTask` launches a cluster with three nodes of bwc version of OpenSearch with alerting and tests backwards compatibility by performing a full restart on the cluster upgrading all the nodes with the current version of OpenSearch with alerting.
+13. `./gradlew bwcTestSuite` runs all the above bwc tests combined.
 
 When launching a cluster using one of the above commands, logs are placed in `alerting/build/testclusters/integTest-0/logs/`. Though the logs are teed to the console, in practices it's best to check the actual log file.
 

diff --git a/alerting/build.gradle b/alerting/build.gradle
@@ -97,6 +97,13 @@ integTest.dependsOn(bundle)
 integTest.getClusters().forEach{c -> c.plugin(project.getObjects().fileProperty().value(bundle.getArchiveFile()))}
 
 def _numNodes = findProperty('numNodes') as Integer ?: 1
+
+String notificationsFilePath = "src/test/resources/notifications"
+String notificationsCoreFilePath = "src/test/resources/notifications-core"
+String notificationsPlugin = "opensearch-notifications-" + plugin_no_snapshot + ".zip"
+String notificationsCorePlugin = "opensearch-notifications-core-" + plugin_no_snapshot + ".zip"
+String notificationsRemoteFile = "https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/" + opensearch_no_snapshot + "/latest/linux/x64/tar/builds/opensearch/plugins/" + notificationsPlugin
+String notificationsCoreRemoteFile = "https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/" + opensearch_no_snapshot + "/latest/linux/x64/tar/builds/opensearch/plugins/" + notificationsCorePlugin
 testClusters.integTest {
     testDistribution = "ARCHIVE"
     // Cluster shrink exception thrown if we try to set numberOfNodes to 1, so only apply if > 1
@@ -115,14 +122,38 @@ testClusters.integTest {
     plugin(provider({
         new RegularFile() {
             @Override
-            File getAsFile() { fileTree("src/test/resources/notifications-core").getSingleFile() }
+            File getAsFile() {
+                File dir = new File(rootDir.path + "/alerting/" + notificationsCoreFilePath)
+
+                if (!dir.exists()) {
+                    dir.mkdirs()
+                }
+
+                File f = new File(dir, notificationsCorePlugin)
+                if (!f.exists()) {
+                    new URL(notificationsCoreRemoteFile).withInputStream{ ins -> f.withOutputStream{ it << ins }}
+                }
+                fileTree(notificationsCoreFilePath).getSingleFile()
+            }
         }
     }))
 
     plugin(provider({
         new RegularFile() {
             @Override
-            File getAsFile() { fileTree("src/test/resources/notifications").getSingleFile() }
+            File getAsFile() {
+                File dir = new File(rootDir.path + "/alerting/" + notificationsFilePath)
+
+                if (!dir.exists()) {
+                    dir.mkdirs()
+                }
+
+                File f = new File(dir, notificationsPlugin)
+                if (!f.exists()) {
+                    new URL(notificationsRemoteFile).withInputStream{ ins -> f.withOutputStream{ it << ins }}
+                }
+                fileTree(notificationsFilePath).getSingleFile()
+            }
         }
     }))
 }
@@ -167,9 +198,36 @@ integTest {
     }
 }
 
+task integTestRemote(type: RestIntegTestTask) {
+    testClassesDirs = sourceSets.test.output.classesDirs
+    classpath = sourceSets.test.runtimeClasspath
+    systemProperty 'tests.security.manager', 'false'
+    systemProperty 'java.io.tmpdir', opensearch_tmp_dir.absolutePath
+
+    systemProperty "https", System.getProperty("https")
+    systemProperty "security", System.getProperty("security")
+    systemProperty "user", System.getProperty("user")
+    systemProperty "password", System.getProperty("password")
+
+    if (System.getProperty("tests.rest.cluster") != null) {
+        filter {
+            includeTestsMatching "org.opensearch.alerting.resthandler.*IT"
+        }
+    }
+
+    if (System.getProperty("https") == null || System.getProperty("https") == "false") {
+        filter {
+            excludeTestsMatching "org.opensearch.alerting.resthandler.Secure*IT"
+        }
+    }
+}
+integTestRemote.enabled = System.getProperty("tests.rest.cluster") != null
+
 String bwcVersion = "1.13.1.0"
 String baseName = "alertingBwcCluster"
 String bwcFilePath = "src/test/resources/bwc"
+String bwcOpenDistroPlugin = "opendistro-alerting-" + bwcVersion + ".zip"
+String bwcRemoteFile = 'https://d3g5vo6xdbdb9a.cloudfront.net/downloads/elasticsearch-plugins/opendistro-alerting/' + bwcOpenDistroPlugin
 
 2.times {i ->
     testClusters {
@@ -183,6 +241,15 @@ String bwcFilePath = "src/test/resources/bwc"
                     return new RegularFile() {
                         @Override
                         File getAsFile() {
+                            File dir = new File(rootDir.path + "/alerting/" + bwcFilePath + "/alerting/" + bwcVersion)
+
+                            if (!dir.exists()) {
+                                dir.mkdirs()
+                            }
+                            File f = new File(dir, bwcOpenDistroPlugin)
+                            if (!f.exists()) {
+                                new URL(bwcRemoteFile).withInputStream{ ins -> f.withOutputStream{ it << ins }}
+                            }
                             return fileTree(bwcFilePath + "/alerting/" + bwcVersion).getSingleFile()
                         }
                     }
@@ -205,13 +272,37 @@ task prepareBwcTests {
                 provider({
                     new RegularFile() {
                         @Override
-                        File getAsFile() { fileTree("src/test/resources/notifications-core").getSingleFile() }
+                        File getAsFile() {
+                            File dir = new File(rootDir.path + "/alerting/" + notificationsCoreFilePath)
+
+                            if (!dir.exists()) {
+                                dir.mkdirs()
+                            }
+
+                            File f = new File(dir, notificationsCorePlugin)
+                            if (!f.exists()) {
+                                new URL(notificationsCoreRemoteFile).withInputStream{ ins -> f.withOutputStream{ it << ins }}
+                            }
+                            fileTree(notificationsCoreFilePath).getSingleFile()
+                        }
                     }
                 }),
                 provider({
                     new RegularFile() {
                         @Override
-                        File getAsFile() { fileTree("src/test/resources/notifications").getSingleFile() }
+                        File getAsFile() {
+                            File dir = new File(rootDir.path + "/alerting/" + notificationsFilePath)
+
+                            if (!dir.exists()) {
+                                dir.mkdirs()
+                            }
+
+                            File f = new File(dir, notificationsPlugin)
+                            if (!f.exists()) {
+                                new URL(notificationsRemoteFile).withInputStream{ ins -> f.withOutputStream{ it << ins }}
+                            }
+                            fileTree(notificationsFilePath).getSingleFile()
+                        }
                     }
                 })
         ]

diff --git a/alerting/src/main/kotlin/org/opensearch/alerting/AlertingPlugin.kt b/alerting/src/main/kotlin/org/opensearch/alerting/AlertingPlugin.kt
@@ -234,7 +234,7 @@ internal class AlertingPlugin : PainlessExtension, ActionPlugin, ScriptPlugin, R
         docLevelMonitorQueries = DocLevelMonitorQueries(client.admin(), clusterService)
         scheduler = JobScheduler(threadPool, runner)
         sweeper = JobSweeper(environment.settings(), client, clusterService, threadPool, xContentRegistry, scheduler, ALERTING_JOB_TYPES)
-        destinationMigrationCoordinator = DestinationMigrationCoordinator(client, clusterService, threadPool)
+        destinationMigrationCoordinator = DestinationMigrationCoordinator(client, clusterService, threadPool, scheduledJobIndices)
         this.threadPool = threadPool
         this.clusterService = clusterService
         return listOf(sweeper, scheduler, runner, scheduledJobIndices, docLevelMonitorQueries, destinationMigrationCoordinator)