[BUG] Dashboards Health check should only run against node which support it. #2203

seraphjiang · 2022-08-24T23:51:43Z

Describe the bug
Dashboards Heath check is not able to exclude node we don't want. e.g. ingest, ml, transform

Expected behavior
Admin should has option to configure Dashboards health check to skip specific node type or specific node attributes.

e.g.

skip ingest node
skip by attribute box_type=warm

// response from 
_cluster/state/nodes
{
      "name" : "cd4f982f2ea88ac087315601e5e09e1a",
      "ephemeral_id" : "oaxei5-",
      "transport_address" : "10.xxx.xxx.39:9300",
      "attributes" : {
        "box_type" : "warm",
        "cold_enabled" : "false",
        "shard_indexing_pressure_enabled" : "true",
      }
    }

OpenSearch Version
1.x, 2.x

Dashboards Version
1.0, 1.1, 1.2, 1.3...
2.0, 2.1

The text was updated successfully, but these errors were encountered:

Enable filtering with custom health checks based on node attributes: ``` opensearch.optimizedHealthcheck.filters: { attribute_key: "attribute_value", } ``` Also, fixes issue that expects the response to array when it was a dictionary. Issue: opensearch-project#2214 opensearch-project#2203 Signed-off-by: Kawika Avilla <kavilla414@gmail.com>

Enable filtering with custom health checks based on node attributes: ``` opensearch.optimizedHealthcheck.filters: { attribute_key: "attribute_value", } ``` Also, fixes issue that expects the response to array when it was a dictionary. Issue: #2214 #2203 Signed-off-by: Kawika Avilla <kavilla414@gmail.com>

Enable filtering with custom health checks based on node attributes: ``` opensearch.optimizedHealthcheck.filters: { attribute_key: "attribute_value", } ``` Also, fixes issue that expects the response to array when it was a dictionary. Issue: #2214 #2203 Signed-off-by: Kawika Avilla <kavilla414@gmail.com> (cherry picked from commit e6bbb40)

Enable filtering with custom health checks based on node attributes: ``` opensearch.optimizedHealthcheck.filters: { attribute_key: "attribute_value", } ``` Also, fixes issue that expects the response to array when it was a dictionary. Issue: opensearch-project#2214 opensearch-project#2203 Signed-off-by: Kawika Avilla <kavilla414@gmail.com>

Original implementation incorrectly assumed the return list of nodes was an object array. This PR: opensearch-project#2232 Addressed the return but didn't catch the nodes.find in the return which is a function for an array. Also, refactors to return a list of node_ids because the original implementation indicated that it should but it can return node ids but it never did. It only returned `null` or `_local`, the problem with this approach is that it doesn't expect valid node version with different DIs or filter out nodes when we pass `null` as the node ID for the node info call because it was fan out the request to all nodes. Now this function will return `_local` if all the nodes share the same cluster_id using a greedy approach since we can assume it is all the same version. Will return node ids, if the cluster_id are different so it will pass a CSV to the node info call and return the info for those nodes. And null if no cluster_id is present, ie, fan out the response. Original issue: opensearch-project#2203 Signed-off-by: Kawika Avilla <kavilla414@gmail.com>

Enable filtering with custom health checks based on node attributes: ``` opensearch.optimizedHealthcheck.filters: { attribute_key: "attribute_value", } ``` Also, fixes issue that expects the response to array when it was a dictionary. Issue: #2214 #2203 Signed-off-by: Kawika Avilla <kavilla414@gmail.com> (cherry picked from commit e6bbb40) Co-authored-by: Kawika Avilla <kavilla414@gmail.com> Co-authored-by: Josh Romero <rmerqg@amazon.com>

Original implementation incorrectly assumed the return list of nodes was an object array. This PR: opensearch-project#2232 Addressed the return but didn't catch the nodes.find in the return which is a function for an array. Also, refactors to return a list of node_ids because the original implementation indicated that it should but it can return node ids but it never did. It only returned `null` or `_local`, the problem with this approach is that it doesn't expect valid node version with different DIs or filter out nodes when we pass `null` as the node ID for the node info call because it was fan out the request to all nodes. Now this function will return `_local` if all the nodes share the same cluster_id using a greedy approach since we can assume it is all the same version. Will return node ids, if the cluster_id are different so it will pass a CSV to the node info call and return the info for those nodes. And null if no cluster_id is present, ie, fan out the response. Original issue: opensearch-project#2203 Signed-off-by: Kawika Avilla <kavilla414@gmail.com>

Original implementation incorrectly assumed the return list of nodes was an object array. This PR: #2232 Addressed the return but didn't catch the nodes.find in the return which is a function for an array. Also, refactors to return a list of node_ids because the original implementation indicated that it should but it can return node ids but it never did. It only returned `null` or `_local`, the problem with this approach is that it doesn't expect valid node version with different DIs or filter out nodes when we pass `null` as the node ID for the node info call because it was fan out the request to all nodes. Now this function will return `_local` if all the nodes share the same cluster_id using a greedy approach since we can assume it is all the same version. Will return node ids, if the cluster_id are different so it will pass a CSV to the node info call and return the info for those nodes. And null if no cluster_id is present, ie, fan out the response. Original issue: #2203 Signed-off-by: Kawika Avilla <kavilla414@gmail.com>

Original implementation incorrectly assumed the return list of nodes was an object array. This PR: #2232 Addressed the return but didn't catch the nodes.find in the return which is a function for an array. Also, refactors to return a list of node_ids because the original implementation indicated that it should but it can return node ids but it never did. It only returned `null` or `_local`, the problem with this approach is that it doesn't expect valid node version with different DIs or filter out nodes when we pass `null` as the node ID for the node info call because it was fan out the request to all nodes. Now this function will return `_local` if all the nodes share the same cluster_id using a greedy approach since we can assume it is all the same version. Will return node ids, if the cluster_id are different so it will pass a CSV to the node info call and return the info for those nodes. And null if no cluster_id is present, ie, fan out the response. Original issue: #2203 Signed-off-by: Kawika Avilla <kavilla414@gmail.com> (cherry picked from commit 3496526)

Enable filtering with custom health checks based on node attributes: ``` opensearch.optimizedHealthcheck.filters: { attribute_key: "attribute_value", } ``` Also, fixes issue that expects the response to array when it was a dictionary. Issue: #2214 #2203 Signed-off-by: Kawika Avilla <kavilla414@gmail.com> (cherry picked from commit e6abd9e)

Enable filtering with custom health checks based on node attributes: ``` opensearch.optimizedHealthcheck.filters: { attribute_key: "attribute_value", } ``` Also, fixes issue that expects the response to array when it was a dictionary. Issue: #2214 #2203 Signed-off-by: Kawika Avilla <kavilla414@gmail.com> (cherry picked from commit e6bbb40) Co-authored-by: Kawika Avilla <kavilla414@gmail.com> Co-authored-by: Josh Romero <rmerqg@amazon.com>

Enable filtering with custom health checks based on node attributes: ``` opensearch.optimizedHealthcheck.filters: { attribute_key: "attribute_value", } ``` Also, fixes issue that expects the response to array when it was a dictionary. Issue: #2214 #2203 Signed-off-by: Kawika Avilla <kavilla414@gmail.com> (cherry picked from commit e6abd9e) Co-authored-by: Kawika Avilla <kavilla414@gmail.com>

Enable filtering with custom health checks based on node attributes: ``` opensearch.optimizedHealthcheck.filters: { attribute_key: "attribute_value", } ``` Also, fixes issue that expects the response to array when it was a dictionary. Issue: #2214 #2203 Signed-off-by: Kawika Avilla <kavilla414@gmail.com> (cherry picked from commit e6bbb40) Co-authored-by: Kawika Avilla <kavilla414@gmail.com>

Original implementation incorrectly assumed the return list of nodes was an object array. This PR: #2232 Addressed the return but didn't catch the nodes.find in the return which is a function for an array. Also, refactors to return a list of node_ids because the original implementation indicated that it should but it can return node ids but it never did. It only returned `null` or `_local`, the problem with this approach is that it doesn't expect valid node version with different DIs or filter out nodes when we pass `null` as the node ID for the node info call because it was fan out the request to all nodes. Now this function will return `_local` if all the nodes share the same cluster_id using a greedy approach since we can assume it is all the same version. Will return node ids, if the cluster_id are different so it will pass a CSV to the node info call and return the info for those nodes. And null if no cluster_id is present, ie, fan out the response. Original issue: #2203 Signed-off-by: Kawika Avilla <kavilla414@gmail.com> (cherry picked from commit 3496526)

…ch-project#2277) Original implementation incorrectly assumed the return list of nodes was an object array. This PR: opensearch-project#2232 Addressed the return but didn't catch the nodes.find in the return which is a function for an array. Also, refactors to return a list of node_ids because the original implementation indicated that it should but it can return node ids but it never did. It only returned `null` or `_local`, the problem with this approach is that it doesn't expect valid node version with different DIs or filter out nodes when we pass `null` as the node ID for the node info call because it was fan out the request to all nodes. Now this function will return `_local` if all the nodes share the same cluster_id using a greedy approach since we can assume it is all the same version. Will return node ids, if the cluster_id are different so it will pass a CSV to the node info call and return the info for those nodes. And null if no cluster_id is present, ie, fan out the response. Original issue: opensearch-project#2203 Signed-off-by: Kawika Avilla <kavilla414@gmail.com>

…2296) Original implementation incorrectly assumed the return list of nodes was an object array. This PR: #2232 Addressed the return but didn't catch the nodes.find in the return which is a function for an array. Also, refactors to return a list of node_ids because the original implementation indicated that it should but it can return node ids but it never did. It only returned `null` or `_local`, the problem with this approach is that it doesn't expect valid node version with different DIs or filter out nodes when we pass `null` as the node ID for the node info call because it was fan out the request to all nodes. Now this function will return `_local` if all the nodes share the same cluster_id using a greedy approach since we can assume it is all the same version. Will return node ids, if the cluster_id are different so it will pass a CSV to the node info call and return the info for those nodes. And null if no cluster_id is present, ie, fan out the response. Original issue: #2203 Signed-off-by: Kawika Avilla <kavilla414@gmail.com> (cherry picked from commit 3496526) Co-authored-by: Kawika Avilla <kavilla414@gmail.com>

…2297) Original implementation incorrectly assumed the return list of nodes was an object array. This PR: #2232 Addressed the return but didn't catch the nodes.find in the return which is a function for an array. Also, refactors to return a list of node_ids because the original implementation indicated that it should but it can return node ids but it never did. It only returned `null` or `_local`, the problem with this approach is that it doesn't expect valid node version with different DIs or filter out nodes when we pass `null` as the node ID for the node info call because it was fan out the request to all nodes. Now this function will return `_local` if all the nodes share the same cluster_id using a greedy approach since we can assume it is all the same version. Will return node ids, if the cluster_id are different so it will pass a CSV to the node info call and return the info for those nodes. And null if no cluster_id is present, ie, fan out the response. Original issue: #2203 Signed-off-by: Kawika Avilla <kavilla414@gmail.com> (cherry picked from commit 3496526) Co-authored-by: Kawika Avilla <kavilla414@gmail.com>

…2300) Original implementation incorrectly assumed the return list of nodes was an object array. This PR: #2232 Addressed the return but didn't catch the nodes.find in the return which is a function for an array. Also, refactors to return a list of node_ids because the original implementation indicated that it should but it can return node ids but it never did. It only returned `null` or `_local`, the problem with this approach is that it doesn't expect valid node version with different DIs or filter out nodes when we pass `null` as the node ID for the node info call because it was fan out the request to all nodes. Now this function will return `_local` if all the nodes share the same cluster_id using a greedy approach since we can assume it is all the same version. Will return node ids, if the cluster_id are different so it will pass a CSV to the node info call and return the info for those nodes. And null if no cluster_id is present, ie, fan out the response. Original issue: #2203 Signed-off-by: Kawika Avilla <kavilla414@gmail.com> Signed-off-by: Kawika Avilla <kavilla414@gmail.com>

…2298) Original implementation incorrectly assumed the return list of nodes was an object array. This PR: #2232 Addressed the return but didn't catch the nodes.find in the return which is a function for an array. Also, refactors to return a list of node_ids because the original implementation indicated that it should but it can return node ids but it never did. It only returned `null` or `_local`, the problem with this approach is that it doesn't expect valid node version with different DIs or filter out nodes when we pass `null` as the node ID for the node info call because it was fan out the request to all nodes. Now this function will return `_local` if all the nodes share the same cluster_id using a greedy approach since we can assume it is all the same version. Will return node ids, if the cluster_id are different so it will pass a CSV to the node info call and return the info for those nodes. And null if no cluster_id is present, ie, fan out the response. Original issue: #2203 Signed-off-by: Kawika Avilla <kavilla414@gmail.com> Signed-off-by: Kawika Avilla <kavilla414@gmail.com>

…2300) Original implementation incorrectly assumed the return list of nodes was an object array. This PR: #2232 Addressed the return but didn't catch the nodes.find in the return which is a function for an array. Also, refactors to return a list of node_ids because the original implementation indicated that it should but it can return node ids but it never did. It only returned `null` or `_local`, the problem with this approach is that it doesn't expect valid node version with different DIs or filter out nodes when we pass `null` as the node ID for the node info call because it was fan out the request to all nodes. Now this function will return `_local` if all the nodes share the same cluster_id using a greedy approach since we can assume it is all the same version. Will return node ids, if the cluster_id are different so it will pass a CSV to the node info call and return the info for those nodes. And null if no cluster_id is present, ie, fan out the response. Original issue: #2203 Signed-off-by: Kawika Avilla <kavilla414@gmail.com> Signed-off-by: Kawika Avilla <kavilla414@gmail.com> (cherry picked from commit e10ba34)

…2300) (#2304) Original implementation incorrectly assumed the return list of nodes was an object array. This PR: #2232 Addressed the return but didn't catch the nodes.find in the return which is a function for an array. Also, refactors to return a list of node_ids because the original implementation indicated that it should but it can return node ids but it never did. It only returned `null` or `_local`, the problem with this approach is that it doesn't expect valid node version with different DIs or filter out nodes when we pass `null` as the node ID for the node info call because it was fan out the request to all nodes. Now this function will return `_local` if all the nodes share the same cluster_id using a greedy approach since we can assume it is all the same version. Will return node ids, if the cluster_id are different so it will pass a CSV to the node info call and return the info for those nodes. And null if no cluster_id is present, ie, fan out the response. Original issue: #2203 Signed-off-by: Kawika Avilla <kavilla414@gmail.com> Signed-off-by: Kawika Avilla <kavilla414@gmail.com> (cherry picked from commit e10ba34) Co-authored-by: Kawika Avilla <kavilla414@gmail.com>

…2300) (#2302) Original implementation incorrectly assumed the return list of nodes was an object array. This PR: #2232 Addressed the return but didn't catch the nodes.find in the return which is a function for an array. Also, refactors to return a list of node_ids because the original implementation indicated that it should but it can return node ids but it never did. It only returned `null` or `_local`, the problem with this approach is that it doesn't expect valid node version with different DIs or filter out nodes when we pass `null` as the node ID for the node info call because it was fan out the request to all nodes. Now this function will return `_local` if all the nodes share the same cluster_id using a greedy approach since we can assume it is all the same version. Will return node ids, if the cluster_id are different so it will pass a CSV to the node info call and return the info for those nodes. And null if no cluster_id is present, ie, fan out the response. Original issue: #2203 Signed-off-by: Kawika Avilla <kavilla414@gmail.com> Signed-off-by: Kawika Avilla <kavilla414@gmail.com> (cherry picked from commit e10ba34) Co-authored-by: Kawika Avilla <kavilla414@gmail.com>

…2300) (#2301) Original implementation incorrectly assumed the return list of nodes was an object array. This PR: #2232 Addressed the return but didn't catch the nodes.find in the return which is a function for an array. Also, refactors to return a list of node_ids because the original implementation indicated that it should but it can return node ids but it never did. It only returned `null` or `_local`, the problem with this approach is that it doesn't expect valid node version with different DIs or filter out nodes when we pass `null` as the node ID for the node info call because it was fan out the request to all nodes. Now this function will return `_local` if all the nodes share the same cluster_id using a greedy approach since we can assume it is all the same version. Will return node ids, if the cluster_id are different so it will pass a CSV to the node info call and return the info for those nodes. And null if no cluster_id is present, ie, fan out the response. Original issue: #2203 Signed-off-by: Kawika Avilla <kavilla414@gmail.com> Signed-off-by: Kawika Avilla <kavilla414@gmail.com> (cherry picked from commit e10ba34) Co-authored-by: Kawika Avilla <kavilla414@gmail.com>

* [BUG] fix healthcheck logic to expect object and return ids (#2277) (#2300) (#2301) Original implementation incorrectly assumed the return list of nodes was an object array. This PR: #2232 Addressed the return but didn't catch the nodes.find in the return which is a function for an array. Also, refactors to return a list of node_ids because the original implementation indicated that it should but it can return node ids but it never did. It only returned `null` or `_local`, the problem with this approach is that it doesn't expect valid node version with different DIs or filter out nodes when we pass `null` as the node ID for the node info call because it was fan out the request to all nodes. Now this function will return `_local` if all the nodes share the same cluster_id using a greedy approach since we can assume it is all the same version. Will return node ids, if the cluster_id are different so it will pass a CSV to the node info call and return the info for those nodes. And null if no cluster_id is present, ie, fan out the response. Original issue: #2203 Signed-off-by: Kawika Avilla <kavilla414@gmail.com> Signed-off-by: Kawika Avilla <kavilla414@gmail.com> (cherry picked from commit e10ba34) Co-authored-by: Kawika Avilla <kavilla414@gmail.com> * change version Signed-off-by: Tao liu <liutaoaz@amazon.com> Signed-off-by: Tao liu <liutaoaz@amazon.com> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: Kawika Avilla <kavilla414@gmail.com>

…2300) (#2303) Original implementation incorrectly assumed the return list of nodes was an object array. This PR: #2232 Addressed the return but didn't catch the nodes.find in the return which is a function for an array. Also, refactors to return a list of node_ids because the original implementation indicated that it should but it can return node ids but it never did. It only returned `null` or `_local`, the problem with this approach is that it doesn't expect valid node version with different DIs or filter out nodes when we pass `null` as the node ID for the node info call because it was fan out the request to all nodes. Now this function will return `_local` if all the nodes share the same cluster_id using a greedy approach since we can assume it is all the same version. Will return node ids, if the cluster_id are different so it will pass a CSV to the node info call and return the info for those nodes. And null if no cluster_id is present, ie, fan out the response. Original issue: #2203 Signed-off-by: Kawika Avilla <kavilla414@gmail.com> Signed-off-by: Kawika Avilla <kavilla414@gmail.com> (cherry picked from commit e10ba34) Co-authored-by: Kawika Avilla <kavilla414@gmail.com>

…ch-project#2277) Original implementation incorrectly assumed the return list of nodes was an object array. This PR: opensearch-project#2232 Addressed the return but didn't catch the nodes.find in the return which is a function for an array. Also, refactors to return a list of node_ids because the original implementation indicated that it should but it can return node ids but it never did. It only returned `null` or `_local`, the problem with this approach is that it doesn't expect valid node version with different DIs or filter out nodes when we pass `null` as the node ID for the node info call because it was fan out the request to all nodes. Now this function will return `_local` if all the nodes share the same cluster_id using a greedy approach since we can assume it is all the same version. Will return node ids, if the cluster_id are different so it will pass a CSV to the node info call and return the info for those nodes. And null if no cluster_id is present, ie, fan out the response. Original issue: opensearch-project#2203 Signed-off-by: Kawika Avilla <kavilla414@gmail.com> Signed-off-by: Sergey V. Osipov <sipopo@yandex.ru>

…-project#2244) Enable filtering with custom health checks based on node attributes: ``` opensearch.optimizedHealthcheck.filters: { attribute_key: "attribute_value", } ``` Also, fixes issue that expects the response to array when it was a dictionary. Issue: opensearch-project#2214 opensearch-project#2203 Signed-off-by: Kawika Avilla <kavilla414@gmail.com> (cherry picked from commit e6bbb40) Co-authored-by: Kawika Avilla <kavilla414@gmail.com>

…ch-project#2277) (opensearch-project#2298) Original implementation incorrectly assumed the return list of nodes was an object array. This PR: opensearch-project#2232 Addressed the return but didn't catch the nodes.find in the return which is a function for an array. Also, refactors to return a list of node_ids because the original implementation indicated that it should but it can return node ids but it never did. It only returned `null` or `_local`, the problem with this approach is that it doesn't expect valid node version with different DIs or filter out nodes when we pass `null` as the node ID for the node info call because it was fan out the request to all nodes. Now this function will return `_local` if all the nodes share the same cluster_id using a greedy approach since we can assume it is all the same version. Will return node ids, if the cluster_id are different so it will pass a CSV to the node info call and return the info for those nodes. And null if no cluster_id is present, ie, fan out the response. Original issue: opensearch-project#2203 Signed-off-by: Kawika Avilla <kavilla414@gmail.com> Signed-off-by: Kawika Avilla <kavilla414@gmail.com>

…ch-project#2277) Original implementation incorrectly assumed the return list of nodes was an object array. This PR: opensearch-project#2232 Addressed the return but didn't catch the nodes.find in the return which is a function for an array. Also, refactors to return a list of node_ids because the original implementation indicated that it should but it can return node ids but it never did. It only returned `null` or `_local`, the problem with this approach is that it doesn't expect valid node version with different DIs or filter out nodes when we pass `null` as the node ID for the node info call because it was fan out the request to all nodes. Now this function will return `_local` if all the nodes share the same cluster_id using a greedy approach since we can assume it is all the same version. Will return node ids, if the cluster_id are different so it will pass a CSV to the node info call and return the info for those nodes. And null if no cluster_id is present, ie, fan out the response. Original issue: opensearch-project#2203 Signed-off-by: Kawika Avilla <kavilla414@gmail.com> Signed-off-by: Sergey V. Osipov <sipopo@yandex.ru>

seraphjiang added bug Something isn't working untriaged labels Aug 24, 2022

kristenTian removed the untriaged label Aug 30, 2022

kristenTian assigned kristenTian and kavilla and unassigned kristenTian Aug 30, 2022

kavilla mentioned this issue Aug 31, 2022

Custom healthcheck with filters #2232

Merged

7 tasks

kavilla linked a pull request Aug 31, 2022 that will close this issue

Custom healthcheck with filters #2232

Merged

7 tasks

kavilla closed this as completed in #2232 Aug 31, 2022

kavilla mentioned this issue Sep 6, 2022

[BUG] fix healthcheck logic to expect object and return ids #2277

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] Dashboards Health check should only run against node which support it. #2203

[BUG] Dashboards Health check should only run against node which support it. #2203

seraphjiang commented Aug 24, 2022 •

edited

Loading

[BUG] Dashboards Health check should only run against node which support it. #2203

[BUG] Dashboards Health check should only run against node which support it. #2203

Comments

seraphjiang commented Aug 24, 2022 • edited Loading

seraphjiang commented Aug 24, 2022 •

edited

Loading