Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

After upgrading to 8.0.0 rc2, error is thrown when editing watchers created in 6.8.23 #83235

Closed
wwang500 opened this issue Jan 27, 2022 · 6 comments · Fixed by #83524
Closed
Assignees
Labels
:ml Machine learning Team:ML Meta label for the ML team v8.0.0

Comments

@wwang500
Copy link

Step to reproduce:

  1. in 6.8.23, create a ml anomaly detection job,
  2. Start the job in real-time and create a watcher for it
  3. upgrade the cluster to 7.17.0 BC3
  4. resolve all critical issues which were reported from 8.0 upgrade assistant
  5. Upgrade it to 8.0.0-rc2 BC2
  6. go to Stack Management, try to edit the watcher

Observed:
Error: Error loading watch could not parse [search] input for watch [ml-job3]. failed to parse [request]

in es.log:

org.elasticsearch.ElasticsearchParseException: could not parse [search] input for watch [ml-job3]. failed to parse [request]
    at org.elasticsearch.xpack.watcher.input.search.SearchInput.parse(SearchInput.java:135) ~[x-pack-watcher-8.0.0-rc2.jar:8.0.0-rc2]
    at org.elasticsearch.xpack.watcher.input.search.SearchInputFactory.parseInput(SearchInputFactory.java:39) ~[x-pack-watcher-8.0.0-rc2.jar:8.0.0-rc2]
    at org.elasticsearch.xpack.watcher.input.search.SearchInputFactory.parseInput(SearchInputFactory.java:20) ~[x-pack-watcher-8.0.0-rc2.jar:8.0.0-rc2]
    at org.elasticsearch.xpack.watcher.input.InputFactory.parseExecutable(InputFactory.java:38) ~[x-pack-watcher-8.0.0-rc2.jar:8.0.0-rc2]
    at org.elasticsearch.xpack.watcher.input.InputRegistry.parse(InputRegistry.java:63) ~[x-pack-watcher-8.0.0-rc2.jar:8.0.0-rc2]
    at org.elasticsearch.xpack.watcher.watch.WatchParser.parse(WatchParser.java:197) ~[x-pack-watcher-8.0.0-rc2.jar:8.0.0-rc2]
    at org.elasticsearch.xpack.watcher.watch.WatchParser.parse(WatchParser.java:168) ~[x-pack-watcher-8.0.0-rc2.jar:8.0.0-rc2]
    at org.elasticsearch.xpack.watcher.watch.WatchParser.parse(WatchParser.java:92) ~[x-pack-watcher-8.0.0-rc2.jar:8.0.0-rc2]
    at org.elasticsearch.xpack.watcher.WatcherService.loadWatches(WatcherService.java:383) [x-pack-watcher-8.0.0-rc2.jar:8.0.0-rc2]
    at org.elasticsearch.xpack.watcher.WatcherService.reloadInner(WatcherService.java:264) [x-pack-watcher-8.0.0-rc2.jar:8.0.0-rc2]
    at org.elasticsearch.xpack.watcher.WatcherService.lambda$reload$1(WatcherService.java:226) [x-pack-watcher-8.0.0-rc2.jar:8.0.0-rc2]
    at org.elasticsearch.xpack.watcher.WatcherService$1.doRun(WatcherService.java:445) [x-pack-watcher-8.0.0-rc2.jar:8.0.0-rc2]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:776) [elasticsearch-8.0.0-rc2.jar:8.0.0-rc2]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-8.0.0-rc2.jar:8.0.0-rc2]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
    at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.elasticsearch.ElasticsearchParseException: could not read search request. unexpected array field [types]
    at org.elasticsearch.xpack.watcher.support.search.WatcherSearchTemplateRequest.fromXContent(WatcherSearchTemplateRequest.java:194) ~[x-pack-watcher-8.0.0-rc2.jar:8.0.0-rc2]
    at org.elasticsearch.xpack.watcher.input.search.SearchInput.parse(SearchInput.java:133) ~[x-pack-watcher-8.0.0-rc2.jar:8.0.0-rc2]
    ... 16 more

the watcher document:

{
        "_index" : ".watches-reindexed-for-8",
        "_id" : "ml-job3",
        "_score" : 1.0,
        "_source" : {
          "trigger" : {
            "schedule" : {
              "interval" : "99s"
            }
          },
          "input" : {
            "search" : {
              "request" : {
                "search_type" : "query_then_fetch",
                "indices" : [
                  ".ml-anomalies-*"
                ],
                "types" : [ ],
                "body" : {
                  "size" : 0,
                  "query" : {
                    "bool" : {
                      "filter" : [
                        {
                          "term" : {
                            "job_id" : "job3"
                          }
                        },
                        {
                          "range" : {
                            "timestamp" : {
                              "gte" : "now-30m"
                            }
                          }
                        },
                        {
                          "terms" : {
                            "result_type" : [
                              "bucket",
                              "record",
                              "influencer"
                            ]
                          }
                        }
                      ]
                    }
                  },
                  "aggs" : {
                    "bucket_results" : {
                      "filter" : {
                        "range" : {
                          "anomaly_score" : {
                            "gte" : 75
                          }
                        }
                      },
                      "aggs" : {
                        "top_bucket_hits" : {
                          "top_hits" : {
                            "sort" : [
                              {
                                "anomaly_score" : {
                                  "order" : "desc"
                                }
                              }
                            ],
                            "_source" : {
                              "includes" : [
                                "job_id",
                                "result_type",
                                "timestamp",
                                "anomaly_score",
                                "is_interim"
                              ]
                            },
                            "size" : 1,
                            "script_fields" : {
                              "start" : {
                                "script" : {
                                  "lang" : "painless",
                                  "source" : "LocalDateTime.ofEpochSecond((doc[\"timestamp\"].date.getMillis()-((doc[\"bucket_span\"].value * 1000)\n * params.padding)) / 1000, 0, ZoneOffset.UTC).toString()+\":00.000Z\"",
                                  "params" : {
                                    "padding" : 10
                                  }
                                }
                              },
                              "end" : {
                                "script" : {
                                  "lang" : "painless",
                                  "source" : "LocalDateTime.ofEpochSecond((doc[\"timestamp\"].date.getMillis()+((doc[\"bucket_span\"].value * 1000)\n * params.padding)) / 1000, 0, ZoneOffset.UTC).toString()+\":00.000Z\"",
                                  "params" : {
                                    "padding" : 10
                                  }
                                }
                              },
                              "timestamp_epoch" : {
                                "script" : {
                                  "lang" : "painless",
                                  "source" : """doc["timestamp"].date.getMillis()/1000"""
                                }
                              },
                              "timestamp_iso8601" : {
                                "script" : {
                                  "lang" : "painless",
                                  "source" : """doc["timestamp"].date"""
                                }
                              },
                              "score" : {
                                "script" : {
                                  "lang" : "painless",
                                  "source" : """Math.round(doc["anomaly_score"].value)"""
                                }
                              }
                            }
                          }
                        }
                      }
                    },
                    "influencer_results" : {
                      "filter" : {
                        "range" : {
                          "influencer_score" : {
                            "gte" : 3
                          }
                        }
                      },
                      "aggs" : {
                        "top_influencer_hits" : {
                          "top_hits" : {
                            "sort" : [
                              {
                                "influencer_score" : {
                                  "order" : "desc"
                                }
                              }
                            ],
                            "_source" : {
                              "includes" : [
                                "result_type",
                                "timestamp",
                                "influencer_field_name",
                                "influencer_field_value",
                                "influencer_score",
                                "isInterim"
                              ]
                            },
                            "size" : 3,
                            "script_fields" : {
                              "score" : {
                                "script" : {
                                  "lang" : "painless",
                                  "source" : """Math.round(doc["influencer_score"].value)"""
                                }
                              }
                            }
                          }
                        }
                      }
                    },
                    "record_results" : {
                      "filter" : {
                        "range" : {
                          "record_score" : {
                            "gte" : 3
                          }
                        }
                      },
                      "aggs" : {
                        "top_record_hits" : {
                          "top_hits" : {
                            "sort" : [
                              {
                                "record_score" : {
                                  "order" : "desc"
                                }
                              }
                            ],
                            "_source" : {
                              "includes" : [
                                "result_type",
                                "timestamp",
                                "record_score",
                                "is_interim",
                                "function",
                                "field_name",
                                "by_field_value",
                                "over_field_value",
                                "partition_field_value"
                              ]
                            },
                            "size" : 3,
                            "script_fields" : {
                              "score" : {
                                "script" : {
                                  "lang" : "painless",
                                  "source" : """Math.round(doc["record_score"].value)"""
                                }
                              }
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          },
          "condition" : {
            "compare" : {
              "ctx.payload.aggregations.bucket_results.doc_count" : {
                "gt" : 0
              }
            }
          },
          "actions" : {
            "log" : {
              "logging" : {
                "level" : "info",
                "text" : "Alert for job [{{ctx.payload.aggregations.bucket_results.top_bucket_hits.hits.hits.0._source.job_id}}] at [{{ctx.payload.aggregations.bucket_results.top_bucket_hits.hits.hits.0.fields.timestamp_iso8601.0}}] score [{{ctx.payload.aggregations.bucket_results.top_bucket_hits.hits.hits.0.fields.score.0}}]"
              }
            }
          },
          "metadata" : {
            "xpack" : {
              "type" : "json"
            }
          },
          "status" : {
            "state" : {
              "active" : true,
              "timestamp" : "2022-01-27T16:27:58.849Z"
            },
            "actions" : {
              "log" : {
                "ack" : {
                  "timestamp" : "2022-01-27T16:27:58.849Z",
                  "state" : "awaits_successful_execution"
                }
              }
            },
            "headers" : {
              "_xpack_security_authentication" : "357zAgAHZWxhc3RpYwEJc3VwZXJ1c2VyCgAAAAEAE2luc3RhbmNlLTAwMDAwMDAwMDAFZm91bmQEZmlsZQAACgA="
            },
            "version" : -1,
            "last_checked" : "2022-01-27T18:16:58.081Z",
            "execution_state" : "execution_not_needed"
          }
        }
      }
@elasticmachine elasticmachine added the Team:Data Management Meta label for data/management team label Jan 27, 2022
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-data-management (Team:Data Management)

@masseyke
Copy link
Member

We don't write anything to the deprecation info API, but we do write it to the deprecation logs. Did you check the deprecation logger critical messages? When I try to reproduce this, I see this in the logs:

[deprecation.elasticsearch][CRITICAL] [types removal] Specifying types in a watcher search request is deprecated.

Or are you saying that it doesn't appear in there either? I didn't know which job you were talking about starting, so I just created a custom watcher with an empty types array while I was in 6.8.

@wwang500
Copy link
Author

@masseyke thank for taking a look. yes, I am seeing [deprecation.elasticsearch][CRITICAL] [types removal] Specifying types in a watcher search request is deprecated. in the deprecations.log

@droberts195
Copy link
Contributor

The deprecation logs are now on a separate page in the upgrade assistant, that is linked from a sentence that says, "If you have application code that calls Elasticsearch APIs, review the Elasticsearch deprecation logs to make sure you are not using deprecated APIs." - see elastic/kibana#118659. This was done because we don't know that a critical deprecation log is something that comes from within the cluster and is bound to still exist after upgrade rather than some external client that might not send its obsolete request again.

So I think there's a UX issue here in that it's easy for a user to miss that one of the deprecation logs among potentially many coming from external clients comes from within the system. With "if you have application code that calls Elasticsearch APIs" we're not making it obvious to the user that the application code in question might live inside the Elasticsearch JVM 😆 .

Since the ML UI in 6.x created watches with a types: [] in their search I am thinking that the friendliest thing to do would be that some periodic check in the ML backend in 7.17 rewrites any searches in watches that contain types: [] so that they don't. In theory this might also rewrite a watch that ML didn't create, but in theory replacing types: [] with nothing in 7.17 shouldn't change behaviour for any watch. We can't get this in in time for 7.17.0, but it could go into a 7.17 patch. Can anyone think of a reason not to do this?

@masseyke masseyke added the Team:ML Meta label for the ML team label Jan 31, 2022
@elasticmachine
Copy link
Collaborator

Pinging @elastic/ml-core (Team:ML)

@masseyke masseyke removed :Data Management/Watcher Team:Data Management Meta label for data/management team labels Jan 31, 2022
@droberts195 droberts195 added the :ml Machine learning label Jan 31, 2022
@droberts195
Copy link
Contributor

droberts195 commented Feb 3, 2022

We discussed this and decided it would be easier and safer for the Watcher code that parses the search in 8.0 and above to ignore the types field if its value is an empty array. This would avoid the risk of trying to update a potentially large number of watch documents. Anything other than an empty array value for types will still be a fatal error.

@droberts195 droberts195 self-assigned this Feb 4, 2022
droberts195 added a commit to droberts195/elasticsearch that referenced this issue Feb 4, 2022
In 6.x internal system components created Watches with an empty
types array in their definition. Types do not exist in 8.x, so
these system-created Watches would need to be modified in 7.x
to remove the types field. Because doing such modifications as
a secret background task could be risky, instead the 8.x Watch
parser will tolerate and ignore an empty types array. It is
clear that an empty types array can be considered identical to
typeless. Non-empty types arrays will still be considered fatal
errors in 8.x, as silently ignoring them could change the meaning
of the search.

Fixes elastic#83235
droberts195 added a commit that referenced this issue Feb 9, 2022
In 6.x internal system components created Watches with an empty
types array in their definition. Types do not exist in 8.x, so
these system-created Watches would need to be modified in 7.x
to remove the types field. Because doing such modifications as
a secret background task could be risky, instead the 8.x Watch
parser will tolerate and ignore an empty types array. It is
clear that an empty types array can be considered identical to
typeless. Non-empty types arrays will still be considered fatal
errors in 8.x, as silently ignoring them could change the meaning
of the search.

Fixes #83235
droberts195 added a commit to droberts195/elasticsearch that referenced this issue Feb 9, 2022
In 6.x internal system components created Watches with an empty
types array in their definition. Types do not exist in 8.x, so
these system-created Watches would need to be modified in 7.x
to remove the types field. Because doing such modifications as
a secret background task could be risky, instead the 8.x Watch
parser will tolerate and ignore an empty types array. It is
clear that an empty types array can be considered identical to
typeless. Non-empty types arrays will still be considered fatal
errors in 8.x, as silently ignoring them could change the meaning
of the search.

Fixes elastic#83235
droberts195 added a commit to droberts195/elasticsearch that referenced this issue Feb 9, 2022
In 6.x internal system components created Watches with an empty
types array in their definition. Types do not exist in 8.x, so
these system-created Watches would need to be modified in 7.x
to remove the types field. Because doing such modifications as
a secret background task could be risky, instead the 8.x Watch
parser will tolerate and ignore an empty types array. It is
clear that an empty types array can be considered identical to
typeless. Non-empty types arrays will still be considered fatal
errors in 8.x, as silently ignoring them could change the meaning
of the search.

Fixes elastic#83235
droberts195 added a commit that referenced this issue Feb 9, 2022
In 6.x internal system components created Watches with an empty
types array in their definition. Types do not exist in 8.x, so
these system-created Watches would need to be modified in 7.x
to remove the types field. Because doing such modifications as
a secret background task could be risky, instead the 8.x Watch
parser will tolerate and ignore an empty types array. It is
clear that an empty types array can be considered identical to
typeless. Non-empty types arrays will still be considered fatal
errors in 8.x, as silently ignoring them could change the meaning
of the search.

Fixes #83235
droberts195 added a commit that referenced this issue Feb 9, 2022
In 6.x internal system components created Watches with an empty
types array in their definition. Types do not exist in 8.x, so
these system-created Watches would need to be modified in 7.x
to remove the types field. Because doing such modifications as
a secret background task could be risky, instead the 8.x Watch
parser will tolerate and ignore an empty types array. It is
clear that an empty types array can be considered identical to
typeless. Non-empty types arrays will still be considered fatal
errors in 8.x, as silently ignoring them could change the meaning
of the search.

Fixes #83235
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:ml Machine learning Team:ML Meta label for the ML team v8.0.0
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants