Implement search alerts tool #1629

ohltyler · 2023-11-14T17:39:22Z

Description

Adds a search alerts tool. Utilizes getAlerts() exposed by the alerting interface in common utils.

Exposes all of the parameters offered by the interface, besides missing since that doesn't seem relevant (defaults to null). All parameters have defaults which match that of the get alerts API for now.

The amount of parameters and/or the default values may be changed later on depending on agent performance and accuracy. For example, too much parameter tuning may lead to messy or invalid results.

Additionally, further integration testing will need to be done to validate different alerting-related questions input to the agent that will have this tool associated to it.

Issues Resolved

Closes #1521

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

ohltyler · 2023-11-14T17:51:58Z

Relevant added tests pass:

./gradlew opensearch-ml-algorithms:test
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 8.1.1
  OS Info               : Linux 5.4.258-178.360.amzn2int.x86_64 (amd64)
  JDK Version           : 14 (OpenJDK)
  JAVA_HOME             : /opt/jdk-14
  Random Testing Seed   : 8F9A68BC7C4A7742
  In FIPS 140 mode      : false
=======================================

> Task :opensearch-ml-common:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: /local/home/ohltyler/workspace/fractal/ml-commons/common/src/main/java/org/opensearch/ml/common/conversation/Interaction.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :opensearch-ml-algorithms:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :opensearch-ml-algorithms:compileTestJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/8.1.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 39s

ml-algorithms/src/main/java/org/opensearch/ml/engine/tools/SearchAlertsTool.java

lezzago · 2023-11-14T21:22:27Z

ml-algorithms/src/main/java/org/opensearch/ml/engine/tools/SearchAlertsTool.java

+        final int tableSize = parameters.containsKey("size") ? Integer.parseInt(parameters.get("size")) : 20;
+        final String searchString = parameters.getOrDefault("searchString", null);
+
+        // not exposing "missing" or "startIndex" from the table, using defaults of null/0, respectively


We would need to expose the start index to paginate the responses.

Is pagination something feasible from the agent? It would need to process and combine multiple alert results after making multiple calls to the tools with different start index values. Is that possible?

Well one call will get a list of alerts. I think this is a good question where we need to understand the limit of data we should be fetching, so we don't overwhelm the LLM. @jngz-es, do you have any recommendations here?

With CoT agent it will be feasible to let LLM/agent to paginate the request to fetch more data. However, as part of tool output it must need to specify if it is a partial or complete output.

I would prefer to make the tool configurable to specify the max result size so that the agent configuration will have flexibility to tweak tool usage based on underlying model.

Max results size will already be covered from tableSize value. I'll change to expose startIndex. My reasoning was to try to limit / scope what we expose and expand later when needed to try to keep things simple.

ml-algorithms/src/main/java/org/opensearch/ml/engine/tools/SearchAlertsTool.java

ylwu-amzn · 2023-11-14T21:48:35Z

ml-algorithms/build.gradle

@@ -47,6 +47,7 @@ dependencies {
    implementation("ai.djl.onnxruntime:onnxruntime-engine:0.21.0") {
        exclude group: "com.microsoft.onnxruntime", module: "onnxruntime"
    }
+    implementation "org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.3.50"


Why need to add this dependency?

For pulling in kotlin code from AlertingPluginInterface

Does just using common-utils not provide this?

api "org.opensearch:common-utils:${common_utils_version}@jar"

@ohltyler Could you please update the response here?

Signed-off-by: Tyler Ohlsen <ohltyler@amazon.com>

ohltyler · 2023-11-15T20:51:11Z

Rebased, confirmed all tests still passing.

ylwu-amzn · 2023-11-15T21:00:19Z

Rebased, confirmed all tests still passing.

@ohltyler , I see Github CI still failed. Have you ran gradlew build locally to verify?

ohltyler · 2023-11-15T22:13:01Z

Rebased, confirmed all tests still passing.

@ohltyler , I see Github CI still failed. Have you ran gradlew build locally to verify?

Yes, relevant tests pass, the error is unrelated to any of these changes. I'm just merging this to the dev branch.

From my understanding this branch is not prioritized to have passing CI, and most changes are being directly merged, which is probably why there are failures. I'm still ensuring these changes aren't causing new failures.

ylwu-amzn · 2023-11-15T22:29:29Z

Check the CI error, and find these

»  org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: codebase property already set: codebase.annotations -> file:/home/runner/work/ml-commons/ml-commons/plugin/build/testclusters/integTest-0/distro/2.11.0-ARCHIVE/plugins/opensearch-ml/annotations-13.0.jar, cannot set to file:/home/runner/work/ml-commons/ml-commons/plugin/build/testclusters/integTest-0/distro/2.11.0-ARCHIVE/plugins/opensearch-ml/annotations-2.21.15.jar

From the error , seems opensearch can't start up. If that's the case, that's not just unit test issue, that mean the OS can't startup, that will block other guys to work on this dev branch.

dbwiddis · 2023-11-15T23:22:45Z

codebase property already set

Could this be from the kotlin dependency (trying to set the codebase as kotlin)? That dependency might need some excludes.

You also have a very old version (1.3.50) of that JAR, latest is 1.9.20

ohltyler · 2023-11-16T00:23:57Z

codebase property already set

Could this be from the kotlin dependency (trying to set the codebase as kotlin)? That dependency might need some excludes.

You also have a very old version (1.3.50) of that JAR, latest is 1.9.20

Thanks for the inputs, confirmed this dependency is causing some problems now, somehow only has started happening after the rebases. I've been testing different versions and using alerting plugin for reference on the kotlin integration. I will push a change after some more testing.

Signed-off-by: Tyler Ohlsen <ohltyler@amazon.com>

ohltyler · 2023-11-16T00:35:52Z

I've changed to 1.8.21 to match that of alerting and ensure this isn't pulling in new/unknown dependencies.

codecov · 2023-11-16T00:50:57Z

Codecov Report

Attention: 13 lines in your changes are missing coverage. Please review.

Comparison is base (39111f9) 67.13% compared to head (8a63aeb) 67.20%.
Report is 1 commits behind head on feature/agent_framework_dev.

Files	Patch %	Lines
...g/opensearch/ml/engine/tools/SearchAlertsTool.java	76.92%	6 Missing and 6 partials ⚠️
...h/ml/action/models/UpdateModelTransportAction.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@                        Coverage Diff                        @@
##             feature/agent_framework_dev    #1629      +/-   ##
=================================================================
+ Coverage                          67.13%   67.20%   +0.06%     
- Complexity                          2488     2496       +8     
=================================================================
  Files                                225      226       +1     
  Lines                              12113    12163      +50     
  Branches                            1249     1257       +8     
=================================================================
+ Hits                                8132     8174      +42     
- Misses                              3436     3437       +1     
- Partials                             545      552       +7

Flag	Coverage Δ
ml-commons	`67.20% <75.47%> (+0.06%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

* Add search alerts tool Signed-off-by: Tyler Ohlsen <ohltyler@amazon.com> * Clean up test params; change validate() Signed-off-by: Tyler Ohlsen <ohltyler@amazon.com> * Add kotlin dependency Signed-off-by: Tyler Ohlsen <ohltyler@amazon.com> * remove spotless Signed-off-by: Tyler Ohlsen <ohltyler@amazon.com> * revert one dep Signed-off-by: Tyler Ohlsen <ohltyler@amazon.com> * Remove clusterservice; simplify param parsing Signed-off-by: Tyler Ohlsen <ohltyler@amazon.com> * Add TYPE; cast o/p to String Signed-off-by: Tyler Ohlsen <ohltyler@amazon.com> * Support null params passed Signed-off-by: Tyler Ohlsen <ohltyler@amazon.com> * Expose startIndex Signed-off-by: Tyler Ohlsen <ohltyler@amazon.com> * Bump kotlin to 1.8.21 Signed-off-by: Tyler Ohlsen <ohltyler@amazon.com> --------- Signed-off-by: Tyler Ohlsen <ohltyler@amazon.com>

ohltyler had a problem deploying to ml-commons-cicd-env November 14, 2023 17:39 — with GitHub Actions Failure

ohltyler had a problem deploying to ml-commons-cicd-env November 14, 2023 17:41 — with GitHub Actions Error

ohltyler had a problem deploying to ml-commons-cicd-env November 14, 2023 17:41 — with GitHub Actions Failure

ohltyler had a problem deploying to ml-commons-cicd-env November 14, 2023 17:41 — with GitHub Actions Error

ohltyler had a problem deploying to ml-commons-cicd-env November 14, 2023 17:41 — with GitHub Actions Failure

arjunkumargiri reviewed Nov 14, 2023

View reviewed changes

ohltyler had a problem deploying to ml-commons-cicd-env November 14, 2023 19:46 — with GitHub Actions Error

ohltyler had a problem deploying to ml-commons-cicd-env November 14, 2023 19:46 — with GitHub Actions Failure

lezzago reviewed Nov 14, 2023

View reviewed changes

dbwiddis reviewed Nov 14, 2023

View reviewed changes

ml-algorithms/src/main/java/org/opensearch/ml/engine/tools/SearchAlertsTool.java Outdated Show resolved Hide resolved

ml-algorithms/src/main/java/org/opensearch/ml/engine/tools/SearchAlertsTool.java Outdated Show resolved Hide resolved

ylwu-amzn reviewed Nov 14, 2023

View reviewed changes

ohltyler force-pushed the search-alerts-tool-2 branch from b8f5c6e to 1641fa3 Compare November 14, 2023 22:45

ohltyler had a problem deploying to ml-commons-cicd-env November 14, 2023 22:45 — with GitHub Actions Error

ohltyler had a problem deploying to ml-commons-cicd-env November 14, 2023 22:45 — with GitHub Actions Failure

ohltyler had a problem deploying to ml-commons-cicd-env November 14, 2023 22:52 — with GitHub Actions Failure

ohltyler had a problem deploying to ml-commons-cicd-env November 14, 2023 22:52 — with GitHub Actions Error

ohltyler had a problem deploying to ml-commons-cicd-env November 14, 2023 22:52 — with GitHub Actions Failure

ohltyler added 5 commits November 15, 2023 20:49

revert one dep

dbb5652

Signed-off-by: Tyler Ohlsen <ohltyler@amazon.com>

Remove clusterservice; simplify param parsing

da31aec

Signed-off-by: Tyler Ohlsen <ohltyler@amazon.com>

Add TYPE; cast o/p to String

49126db

Signed-off-by: Tyler Ohlsen <ohltyler@amazon.com>

Support null params passed

4d0622e

Signed-off-by: Tyler Ohlsen <ohltyler@amazon.com>

Expose startIndex

455ec8e

Signed-off-by: Tyler Ohlsen <ohltyler@amazon.com>

ohltyler force-pushed the search-alerts-tool-2 branch from 9689368 to 455ec8e Compare November 15, 2023 20:50

ohltyler had a problem deploying to ml-commons-cicd-env November 15, 2023 20:50 — with GitHub Actions Failure

ohltyler had a problem deploying to ml-commons-cicd-env November 15, 2023 20:50 — with GitHub Actions Error

ohltyler had a problem deploying to ml-commons-cicd-env November 15, 2023 20:50 — with GitHub Actions Failure

Bump kotlin to 1.8.21

8a63aeb

Signed-off-by: Tyler Ohlsen <ohltyler@amazon.com>

ohltyler temporarily deployed to ml-commons-cicd-env November 16, 2023 00:33 — with GitHub Actions Inactive

ohltyler had a problem deploying to ml-commons-cicd-env November 16, 2023 00:33 — with GitHub Actions Failure

ohltyler temporarily deployed to ml-commons-cicd-env November 16, 2023 00:34 — with GitHub Actions Inactive

ohltyler had a problem deploying to ml-commons-cicd-env November 16, 2023 00:34 — with GitHub Actions Error

ylwu-amzn approved these changes Nov 16, 2023

View reviewed changes

ylwu-amzn merged commit 742f920 into opensearch-project:feature/agent_framework_dev Nov 16, 2023
4 of 7 checks passed

ohltyler deleted the search-alerts-tool-2 branch November 16, 2023 17:26

ohltyler mentioned this pull request Nov 27, 2023

[FEATURE] Tool development for Search Alerts Tool #1521

Closed

lezzago mentioned this pull request Nov 30, 2023

[RFC] OpenSearch Assistant Toolkit opensearch-project/dashboards-assistant#18

Open

ohltyler mentioned this pull request Dec 11, 2023

[Frontport main] Implement search alerts tool #1748

Closed

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement search alerts tool #1629

Implement search alerts tool #1629

ohltyler commented Nov 14, 2023 •

edited

Loading

ohltyler commented Nov 14, 2023

lezzago Nov 14, 2023

ohltyler Nov 14, 2023

lezzago Nov 15, 2023

arjunkumargiri Nov 15, 2023

ohltyler Nov 15, 2023

ohltyler Nov 15, 2023

ylwu-amzn Nov 14, 2023

ohltyler Nov 14, 2023

dbwiddis Nov 15, 2023

dhrubo-os Dec 18, 2023

ohltyler commented Nov 15, 2023

ylwu-amzn commented Nov 15, 2023

ohltyler commented Nov 15, 2023 •

edited

Loading

ylwu-amzn commented Nov 15, 2023 •

edited

Loading

dbwiddis commented Nov 15, 2023

ohltyler commented Nov 16, 2023

ohltyler commented Nov 16, 2023

codecov bot commented Nov 16, 2023

Implement search alerts tool #1629

Implement search alerts tool #1629

Conversation

ohltyler commented Nov 14, 2023 • edited Loading

Description

Issues Resolved

Check List

ohltyler commented Nov 14, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ohltyler commented Nov 15, 2023

ylwu-amzn commented Nov 15, 2023

ohltyler commented Nov 15, 2023 • edited Loading

ylwu-amzn commented Nov 15, 2023 • edited Loading

dbwiddis commented Nov 15, 2023

ohltyler commented Nov 16, 2023

ohltyler commented Nov 16, 2023

codecov bot commented Nov 16, 2023

Codecov Report

ohltyler commented Nov 14, 2023 •

edited

Loading

ohltyler commented Nov 15, 2023 •

edited

Loading

ylwu-amzn commented Nov 15, 2023 •

edited

Loading