Failing ES Promotion: test/functional/apps/getting_started/_shakespeare·js.Getting Started Shakespeare should configure Terms aggregation on play_name

[brianseeders]

This failing test is preventing the promotion of a new 7.11 and master snapshot of ES.

[00:17:28]               └- ✖ fail: Getting Started  Shakespeare should configure Terms aggregation on play_name
[00:17:28]               │      retry.try timeout: Error: expected [ 65, 62, 55, 53, 51 ] to sort of equal [ 71, 65, 62, 55, 55 ]
[00:17:28]               │     at Assertion.assert (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:100:11)
[00:17:28]               │     at Assertion.eql (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:244:8)
[00:17:28]               │     at retry.try (/dev/shm/workspace/kibana/test/functional/apps/getting_started/_shakespeare.js:139:25)
[00:17:28]               │     at process._tickCallback (internal/process/next_tick.js:68:7)

See https://kibana-ci.elastic.co/job/elasticsearch+snapshots+verify/1687/testReport/junit/Chrome%20UI%20Functional%20Tests/test_functional_apps_getting_started__shakespeare%C2%B7js/Kibana_Pipeline___kibana_oss_agent___Getting_Started__Shakespeare_should_configure_Terms_aggregation_on_play_name/

[elasticmachine]

Pinging @elastic/kibana-qa (Team:QA)

[LeeDr]

This seems like an actual Elasticsearch bug to me. Either that or there's something wrong with our query.

The play_name in this data which has the expected unique count of 71 is Richard III. It should be the first bar in this vertical bar graph but doesn't appear in the results at all until I change the terms agg size up to 12 (from the default of 5).

When the size for the terms agg is less than 12, the largest value that appears is Henry VI Part 2 with the value of 65 as shown in the error when the test fails. I pinged the Elasticsearch team and I'll wait to see their response before filing an issue and I'll reference it here.

The next largest is 62, then 55 which tracks the offset in the expected values. But there's more differences after that.

As another experiment I used size: 2 and size: 1 and also got very confusing results.

Here's the first request (copied from the visualization Inspect menu) with size: 2;

{
  "aggs": {
    "2": {
      "terms": {
        "field": "play_name",
        "order": {
          "1": "desc"
        },
        "size": 2
      },
      "aggs": {
        "1": {
          "cardinality": {
            "field": "speaker"
          }
        }
      }
    }
  },
  "size": 0,
  "fields": [],
  "script_fields": {},
  "stored_fields": [
    "*"
  ],
  "_source": {
    "excludes": []
  },
  "query": {
    "bool": {
      "must": [],
      "filter": [
        {
          "match_all": {}
        }
      ],
      "should": [],
      "must_not": []
    }
  }
}

The response when `size: 2` is "Henry VI Part 2" with 65, and "Coriolanus" with 62
{
  "took": 4,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 111396,
    "max_score": null,
    "hits": []
  },
  "aggregations": {
    "2": {
      "doc_count_error_upper_bound": -1,
      "sum_other_doc_count": 104070, << this seems wrong. There are 111,396 docs (107404 + 3992)
      "buckets": [
        {
          "1": {
            "value": 65
          },
          "key": "Henry VI Part 2",
          "doc_count": 3334
        },
        {
          "1": {
            "value": 62
          },
          "key": "Coriolanus",
          "doc_count": 3992
        }
      ]
    }
  }
}

Changed to only get size: 1

{
  "aggs": {
    "2": {
      "terms": {
        "field": "play_name",
        "order": {
          "1": "desc"
        },
        "size": 1
      },
      "aggs": {
        "1": {
          "cardinality": {
            "field": "speaker"
          }
        }
      }
    }
  },
  "size": 0,
  "fields": [],
  "script_fields": {},
  "stored_fields": [
    "*"
  ],
  "_source": {
    "excludes": []
  },
  "query": {
    "bool": {
      "must": [],
      "filter": [
        {
          "match_all": {}
        }
      ],
      "should": [],
      "must_not": []
    }
  }
}

The response when size: 1 should show "Henry VI Part 2" with 65, but instead shows "Coriolanus" with 62
(and in fact, it should show Richard III with 71)

{
  "took": 3,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 111396,
    "max_score": null,
    "hits": []
  },
  "aggregations": {
    "2": {
      "doc_count_error_upper_bound": -1,
      "sum_other_doc_count": 107404, << There are 111,396 docs (107404 + 3992)
      "buckets": [
        {
          "1": {
            "value": 62
          },
          "key": "Coriolanus",
          "doc_count": 3992
        }
      ]
    }
  }
}

[LeeDr]

Elasticsearch team has a PR for a fix. Once it's merged I'll unskip this Kibana test.