Render Mappings more Compact in GET /_cluster/state

[original-brownbear]

This deduplicates index mappings in the /_cluster/state response by grouping them by the sha256 hash of the mapping.
This significantly reduces the response size for large cluster states,
making this debug API easier to read for humans (because understanding
whether or not mappings are duplicated across indices might be valuable)
and more importantly less ressource hungry as far as serialization goes.

Adds an undocumented parameter to get back to the old format if we actually have to in an unforeseen instance.

New rendering of an index:

      "my-index-000001" : {
        "version" : 5,
        "mapping_version" : 2,
        "settings_version" : 1,
        "aliases_version" : 1,
        "routing_num_shards" : 1024,
        "state" : "open",
        "settings" : {
          "index" : {
            "routing" : {
              "allocation" : {
                "include" : {
                  "_tier_preference" : "data_content"
                }
              }
            },
            "number_of_shards" : "1",
            "provided_name" : "my-index-000001",
            "creation_date" : "1644601309989",
            "number_of_replicas" : "1",
            "uuid" : "9m19TP4DRsevoKkSG3DC0g",
            "version" : {
              "created" : "8020099"
            }
          }
        },
        "mappings" : "XinOY78Sl3l55aaFJrhIKDlniKVodszNHRmKhONrR/o=",

and then mappings at the same level as "indices":

  "mappings" : {
      "5jySc/VOXLKC1itGKeilXwF6kCFL0i5Xe0DqJ0eDgwk=" : {
        "dynamic" : "strict",
        "_meta" : {
          "version" : "8.2.0"
        },
        "properties" : {
          "chunk" : {
            "type" : "integer"
          },
          "data" : {
            "type" : "binary"
          },
          "name" : {
            "type" : "keyword"
          }
        }
      },
      "XinOY78Sl3l55aaFJrhIKDlniKVodszNHRmKhONrR/o=" : {
        "properties" : {
          "@timestamp" : {
            "type" : "date"
          },
          "message" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "user" : {
            "properties" : {
              "id" : {
                "type" : "text",
                "fields" : {
                  "keyword" : {
                    "type" : "keyword",
                    "ignore_above" : 256
                  }
                }
              }
            }
          }
        }
      }
    }

relates #77466

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[elasticsearchmachine]

Hi @original-brownbear, I've created a changelog YAML for you.

[DaveCTurner]

Generally +1 from me but we still need to consider whether to treat this as a breaking change or not. We could alternatively not change the default but start sending 429s to clients that request an unreasonably large response. Marking this as Request changes until these questions are resolved.

[henningandersen]

Preparing for our discussion today, I read through this and provided a couple of comments.

[henningandersen]

I would prefer to pick a new name for the SHA256. mappings_sha256 maybe or perhaps shared_mappings_id. Then we can also allow dumping both, which could provide a transitional path for clients.

[henningandersen]

Perhaps we can call this shared_mappings and then have a flag of the same name to the API as well as a cluster setting to disable dumping non-shared mappings?

[DaveCTurner]

Henning and I discussed this again today. We're not confident enough that nobody would be affected by this change in the default behaviour, and we think with chunked-encoding we can protect Elasticsearch against harm at least, so we would prefer to make the desired behaviour opt-in today, deprecate the old default, and remove it in the next major version.

I'll raise a proposal to do this with the breaking changes committee.

[elasticsearchmachine]

Pinging @elastic/es-distributed-obsolete (Team:Distributed (Obsolete))

[elasticsearchmachine]

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)