New Monitoring tests are failing

[elasticmachine]

Original comment by @ywelsch:

Relates LINK REDACTED

See LINK REDACTED

Relevant log lines:

08:12:52   1> [2017-11-10T02:12:51,177][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] [test {p0=old_cluster/60_monitoring/Index monitoring data and search on the old cluster}]: after test
08:12:52   1> [2017-11-10T02:12:51,179][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] Stash dump on failure [{
08:12:52   1>   "stash" : {
08:12:52   1>     "body" : {
08:12:52   1>       "took" : 0,
08:12:52   1>       "timed_out" : false,
08:12:52   1>       "_shards" : {
08:12:52   1>         "total" : 0,
08:12:52   1>         "successful" : 0,
08:12:52   1>         "skipped" : 0,
08:12:52   1>         "failed" : 0
08:12:52   1>       },
08:12:52   1>       "hits" : {
08:12:52   1>         "total" : 0,
08:12:52   2> REPRODUCE WITH: gradle :x-pack-elasticsearch:qa:rolling-upgrade:with-system-key:v5.6.5-SNAPSHOT#oldClusterTestRunner -Dtests.seed=74B035F2BA325802 -Dtests.class=org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT -Dtests.method="test {p0=old_cluster/60_monitoring/Index monitoring data and search on the old cluster}" -Dtests.security.manager=true -Dtests.locale=ar-AE -Dtests.timezone=Canada/Saskatchewan -Dtests.rest.suite=old_cluster
08:12:52   1>         "max_score" : 0.0,
08:12:52   1>         "hits" : [ ]
08:12:52   1>       }
08:12:52   1>     }
08:12:52   1>   }
08:12:52   1> }]
08:12:52 FAILURE 30.3s | UpgradeClusterClientYamlTestSuiteIT.test {p0=old_cluster/60_monitoring/Index monitoring data and search on the old cluster} <<< FAILURES!
08:12:52    > Throwable LINK REDACTED: java.lang.AssertionError: Failure at [old_cluster/60_monitoring:30]: hits.total didn't match expected value:
08:12:52    >                     hits.total: expected [2] but was [0]
08:12:52    > 	at __randomizedtesting.SeedInfo.seed([74B035F2BA325802:FCE40A2814CE35FA]:0)
08:12:52    > 	at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:352)
08:12:52    > 	at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:332)
08:12:52    > 	at java.lang.Thread.run(Thread.java:748)
08:12:52    > Caused by: java.lang.AssertionError: hits.total didn't match expected value:
08:12:52    >                     hits.total: expected [2] but was [0]
08:12:52    > 	at org.elasticsearch.test.rest.yaml.section.MatchAssertion.doAssert(MatchAssertion.java:87)
08:12:52    > 	at org.elasticsearch.test.rest.yaml.section.Assertion.execute(Assertion.java:76)
08:12:52    > 	at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:348)
08:12:52    > 	... 38 more

It looks to me as the test has no condition to wait for the .monitoring-kibana-* to be created.

i.e. something like

 - do:
      cluster.health:
        index: ".monitoring-kibana-*"
        wait_for_active_shards: 2 # 1 primary and 1 replica since we have two nodes

[elasticmachine]

Original comment by @tlrx:

The test failed again today:
LINK REDACTED

with the error:

UpgradeClusterClientYamlTestSuiteIT.test {p0=upgraded_cluster/60_monitoring/Index monitoring data and search on the upgraded cluster} <<< FAILURES!
   > Throwable LINK REDACTED: java.lang.AssertionError: Failure at [upgraded_cluster/60_monitoring:46]: hits.total didn't match expected value:
   >                     hits.total: expected [3] but was [0]
   > 	at __randomizedtesting.SeedInfo.seed([9AC76E423C873DCC:12935198927B5034]:0)
   > 	at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:352)
   > 	at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:332)
   > 	at java.lang.Thread.run(Thread.java:748)
   > Caused by: java.lang.AssertionError: hits.total didn't match expected value:
   >                     hits.total: expected [3] but was [0]
   > 	at org.elasticsearch.test.rest.yaml.section.MatchAssertion.doAssert(MatchAssertion.java:87)
   > 	at org.elasticsearch.test.rest.yaml.section.Assertion.execute(Assertion.java:76)
   > 	at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:348)
   > 	... 38 more

The test execution log is LINK REDACTED

[elasticmachine]

Original comment by @tlrx:

And again: LINK REDACTED

LINK REDACTED

[elasticmachine]

Original comment by @tlrx:

And again LINK REDACTED
LINK REDACTED

LINK REDACTED
LINK REDACTED

[elasticmachine]

Original comment by @polyfractal:

Some more from today:

• LINK REDACTED
• LINK REDACTED
• LINK REDACTED

I'm going to add a skip to this test so it stops causing noise until it is fixed.

[elasticmachine]

Original comment by @spinscale:

@pickypg can you take a look?

I am rather confused by the logging output. There is no mention of xpack.monitoring.bulk being called in the logs, even though that action is called as part of that tests. Hopefully I am missing something...

[elasticmachine]

Original comment by @imotov:

Failed again in 6.0. I cherry-picked @spinscale's commit and pushed it into 6.0

[elasticmachine]

Pinging @elastic/es-core-infra