Rollup Index Pattern creation fails when the amount of fields generated by the rollup job exceeds 32KB

[lucabelluccini]

Kibana version: 7.8.x and below

Elasticsearch version: aligned with Kibana

Describe the bug:

When creating a Rollup Index Pattern with an amount of fields / aggregations which exceeds 32KB, it fails with a "Bad request" error.

Checking the HAR file, we can see there's an error:

bytes can be at most 32766 in length; got 36444: [illegal_argument_exception] Document contains at least one immense term in field=\\\"index-pattern.typeMeta\\\" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: (....)

Steps to reproduce:

1. Create a Rollup Job for the metricbeat indices and add as much fields as you can both in metrics & histogram. Use the rolledup-metricbeat as target index
2. Create a Rollup Index Pattern to cover rolledup-metricbeat,metricbeat* indices
3. It should fail when saving the index template

Expected behavior:

The index template should be created.

Any additional context:

This issue describes a consequence of #70813

Workaround:

TO BE CONFIRMED

A workaround for this issue would be to add the following mapping to the Kibana index:

PUT .kibana/_mapping
{
  "properties": {
    "index-pattern": {
      "properties": {
        "typeMeta": {
          "type": "keyword",
          "ignore_above": 2048
        }
      }
    }
  }
}

BUT BE AWARE THIS MIGHT NOT BE ENOUGH: when Kibana is upgraded, the Kibana index gets reindexed to add new features. The new index will not have this mapping and it might fail the migration of the Saved Objects.

Until a fix is released, it is necessary to add an index template which should prevent index migration errors.

The index template MUST be deleted prior upgrading Kibana to the version which includes the fix.

PUT _template/kibana_workaround
{
  "index_patterns": [ ".kibana", ".kibana_1*", ".kibana_2*", ".kibana_3*", ".kibana_4*", ".kibana_5*", ".kibana_6*", ".kibana_7*", ".kibana_8*", ".kibana_9*" ],
  "mappings": {
    "properties": {
      "index-pattern": {
        "properties": {
          "typeMeta": {
            "type": "keyword",
            "ignore_above": 2048
          }
        }
      }
    }
  }
}

Prior to upgrading to Kibana 7.10, delete the template using DELETE _template/kibana_workaround

[elasticmachine]

Pinging @elastic/kibana-app-arch (Team:AppArch)

[mattkime]

addressed by #74817 - will be released in 7.10

Remember to do this -

The index template should be deleted prior upgrading Kibana to the version which includes the fix.