TokenBackwardsCompatibilityIT fails on 6.x

[javanna]

TokenBackwardsCompatibilityIT#testTokenWorksInMixedOrUpgradedCluster fails on 6.x.

The reproduce line does not quite work (the test gets skipped for me due to a failling assumption), but I was able to reproduce with the same seed executing from x-pack/qa/rolling-upgrade/with-system-key the following command: ../../../../gradlew check -Dtests.security.manager=true -Dtests.locale=nl-BE -Dtests.timezone=Antarctica/South_Pole -Dcompiler.java=10 -Druntime.java=8 -Dtests.seed=142E5A90225DE2B9.

Example of failure: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+bwc-tests/1117

08:15:01 ERROR   0.20s | TokenBackwardsCompatibilityIT.testTokenWorksInMixedOrUpgradedCluster <<< FAILURES!
08:15:01    > Throwable #1: org.elasticsearch.client.ResponseException: method [GET], host [http://[::1]:42832], URI [_xpack/security/_authenticate], status line [HTTP/1.1 401 Unauthorized]
08:15:01    > {"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication token for REST request [_xpack/security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\""]}}],"type":"security_exception","reason":"missing authentication token for REST request [_xpack/security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\""]}},"status":401}
08:15:01    > 	at __randomizedtesting.SeedInfo.seed([142E5A90225DE2B9:6D2D8576BBC9E8A8]:0)
08:15:01    > 	at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:920)
08:15:01    > 	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:227)
08:15:01    > 	at org.elasticsearch.upgrades.TokenBackwardsCompatibilityIT.assertTokenWorks(TokenBackwardsCompatibilityIT.java:185)
08:15:01    > 	at org.elasticsearch.upgrades.TokenBackwardsCompatibilityIT.testTokenWorksInMixedOrUpgradedCluster(TokenBackwardsCompatibilityIT.java:70)
08:15:01    > 	at java.lang.Thread.run(Thread.java:748)
08:15:01    > Caused by: org.elasticsearch.client.ResponseException: method [GET], host [http://[::1]:42832], URI [_xpack/security/_authenticate], status line [HTTP/1.1 401 Unauthorized]
08:15:01    > {"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication token for REST request [_xpack/security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\""]}}],"type":"security_exception","reason":"missing authentication token for REST request [_xpack/security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\""]}},"status":401}

It seems like this is a 6.x only failure and it started fairly recently, in the last days.

Note that another test seems to be failing whenever this test fails: -Dtests.class=org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT \ -Dtests.method="test {p0=mixed_cluster/50_token_auth/Get the indexed token and use if to authenticate}" , which seems like the same failure.

 expected [2xx] status code but api [xpack.security.authenticate] returned [401 Unauthorized] [{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication token for REST request [/_xpack/security/_authenticate?error_trace=true]"

[elasticmachine]

Pinging @elastic/es-security

[jaymode]

Sorry for the noise. This is due to my bad backport yesterday. I pushed d9044c7