[Eleasticsearch] Remove hidden field from indice metricset docs

[crespocarlos]

Proposed commit message

This PR removes the hidden flag introduced by #18639. The SM UI doesn't use this flag, and retrieving which indices are hidden is an expensive operation that can sometimes cause problems such as timeouts depending on how large a cluster is.

elasticsearch": {
  "cluster": {
    "id": "5QYQ1YeeRjy9MJtahqzY6w",
    "name": "elasticsearch"
  },
  "index": {
    "status": "green",
    "hidden": false,
    "shards": {
      "total": 1,
      "primaries": 1
    }

   ...
  }
  ...
}

"elasticsearch": {
  "cluster": {
    "id": "5QYQ1YeeRjy9MJtahqzY6w",
    "name": "elasticsearch"
  },
  "index": {
    "status": "green",
    "shards": {
      "total": 1,
      "primaries": 1
    }
   ...
  }
 ...
}

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

• Start a local ES and kibana instances
• Start metricbeat with elasticsearch module enabled
• Run the query below and check if there is no hidden field in the index object

GET .monitoring-es-*,metrics-*/_search
{
  "size": 5,
  "query": {
    "bool": {
      "filter": [
        {
          "bool": {
            "should": [
              {
                "term": {
                  "data_stream.dataset": "elasticsearch.stack_monitoring.index"
                }
              },
              {
                "term": {
                  "metricset.name": "index"
                }
              },
              {
                "term": {
                  "type": "index_stats"
                }
              }
            ]
          }
        },
        {
          "range": {
            "timestamp": {
              "gte": "now-15m",
              "lte": "now"
            }
          }
        }
      ]
    }
  },
  "collapse": {
    "field": "index_stats.index",
    "inner_hits": {
      "name": "earliest",
      "size": 1,
      "sort": [
        {
          "timestamp": {
            "order": "asc",
            "unmapped_type": "long"
          }
        }
      ]
    }
  },
  "sort": [
    {
      "timestamp": {
        "order": "desc",
        "unmapped_type": "long"
      }
    }
  ]
}

Related issues

closes #39161

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @crespocarlos? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2024-04-25T01:42:06.842+0000

• Duration: 126 min 38 sec

Test stats 🧪

Test	Results
Failed	0
Passed	4625
Skipped	915
Total	5540

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[klacabane]

was this only used by hidden flag ?

[crespocarlos]

yes. I couldn't find other usages of that function

[klacabane]

LGTM