Failing ES Promotion: apis Fleet Endpoints fleet_unenroll_agent allow to unenroll using a list of ids

[mistic]

This failure is preventing the promotion of the current Elasticsearch nightly snapshot.

7.x/7.9: https://kibana-ci.elastic.co/job/elasticsearch+snapshots+verify/856/
7.8/7.8.1: https://kibana-ci.elastic.co/job/elasticsearch+snapshots+verify/867/

For more information on the Elasticsearch snapshot promotion process: https://www.elastic.co/guide/en/kibana/master/development-es-snapshots.html

Failed Tests Reporter:
  - Test has failed 5 times on tracked branches: https://github.com/elastic/kibana/issues/64696

[00:00:00]       │
[00:00:00]         └-: apis
[00:00:00]           └-> "before all" hook
[00:09:38]           └-: Fleet Endpoints
[00:09:38]             └-> "before all" hook
[00:09:46]             └-: fleet_unenroll_agent
[00:09:46]               └-> "before all" hook
[00:09:46]               └-> "before all" hook
[00:09:46]                 │ info [fleet/agents] Loading "mappings.json"
[00:09:46]                 │ info [fleet/agents] Loading "data.json"
[00:09:46]                 │ info [o.e.c.m.MetadataDeleteIndexService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] [.kibana_1/PMYsshv5SYK_xvDJA34J-Q] deleting index
[00:09:46]                 │ info [o.e.c.m.MetadataDeleteIndexService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] [.kibana_2/uj8xOfn-Rye46QUprYqzPQ] deleting index
[00:09:46]                 │ info [fleet/agents] Deleted existing index [".kibana_2",".kibana_1"]
[00:09:46]                 │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] [.kibana_1] creating index, cause [api], templates [], shards [1]/[0], mappings [_doc]
[00:09:46]                 │ info [fleet/agents] Created index ".kibana_1"
[00:09:46]                 │ debg [fleet/agents] ".kibana_1" settings {"index":{"auto_expand_replicas":"0-1","number_of_replicas":"0","number_of_shards":"1"}}
[00:09:46]                 │ info [fleet/agents] Indexed 12 docs into ".kibana"
[00:09:46]                 │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] [.kibana_1/QUf4g-qcQ_6RTCGZagkORg] update_mapping [_doc]
[00:09:46]                 │ debg Migrating saved objects
[00:09:47]                 │ proc [kibana]   log   [20:44:25.122] [info][savedobjects-service] Creating index .kibana_2.
[00:09:47]                 │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] [.kibana_2] creating index, cause [api], templates [], shards [1]/[1], mappings [_doc]
[00:09:47]                 │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] updating number_of_replicas to [0] for indices [.kibana_2]
[00:09:47]                 │ proc [kibana]   log   [20:44:25.228] [info][savedobjects-service] Migrating .kibana_1 saved objects to .kibana_2
[00:09:47]                 │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] [.kibana_2/iNHtlHXuTc2NHsAlDAKFZQ] update_mapping [_doc]
[00:09:47]                 │ proc [kibana]   log   [20:44:25.305] [info][savedobjects-service] Pointing alias .kibana to .kibana_2.
[00:09:47]                 │ proc [kibana]   log   [20:44:25.378] [info][savedobjects-service] Finished in 259ms.
[00:09:47]               └-> "before all" hook
[00:09:47]                 │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] [.kibana_2/iNHtlHXuTc2NHsAlDAKFZQ] update_mapping [_doc]
[00:09:47]                 │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] [events-index_pattern_placeholder] creating index, cause [api], templates [], shards [1]/[1], mappings [_doc]
[00:09:47]                 │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] [logs-index_pattern_placeholder] creating index, cause [api], templates [], shards [1]/[1], mappings [_doc]
[00:09:48]                 │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] [metrics-index_pattern_placeholder] creating index, cause [api], templates [], shards [1]/[1], mappings [_doc]
[00:09:48]                 │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] [.kibana_2/iNHtlHXuTc2NHsAlDAKFZQ] update_mapping [_doc]
[00:09:48]                 │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] [.kibana_2/iNHtlHXuTc2NHsAlDAKFZQ] update_mapping [_doc]
[00:09:48]                 │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] [.kibana_2/iNHtlHXuTc2NHsAlDAKFZQ] update_mapping [_doc]
[00:09:49]                 │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] [.kibana_2/iNHtlHXuTc2NHsAlDAKFZQ] update_mapping [_doc]
[00:09:49]                 │ proc [kibana]   log   [20:44:26.765] [error][ingestManager][plugins] Error connecting to package registry at https://epr.elastic.co/epr/base/base-0.2.0.tar.gz: Error connecting to package registry at https://epr.elastic.co/epr/base/base-0.2.0.tar.gz: Backend unavailable, connection timeout
[00:09:49]                 │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index template [metrics-system.cpu]
[00:09:49]                 │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index template [metrics-system.core]
[00:09:50]                 │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index template [logs-system.auth]
[00:09:50]                 │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index template [metrics-system.diskio]
[00:09:50]                 │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index template [metrics-system.entropy]
[00:09:50]                 │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index template [metrics-system.memory]
[00:09:50]                 │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index template [metrics-system.load]
[00:09:50]                 │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index template [metrics-system.filesystem]
[00:09:50]                 │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index template [metrics-system.fsstat]
[00:09:50]                 │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index template [metrics-system.raid]
[00:09:50]                 │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index template [metrics-system.service]
[00:09:50]                 │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index template [metrics-system.socket]
[00:09:50]                 │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index template [metrics-system.network_summary]
[00:09:50]                 │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index template [metrics-system.process]
[00:09:50]                 │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index template [metrics-system.process_summary]
[00:09:50]                 │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index template [metrics-system.network]
[00:09:50]                 │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index template [logs-system.syslog]
[00:09:50]               └-> allow to unenroll using a list of ids
[00:09:50]                 └-> "before each" hook: global before each
[00:09:50]                 └-> "before each" hook
[00:09:50]                   │ proc [kibana]  error  [20:44:26.816]  Error: Internal Server Error
[00:09:50]                   │ proc [kibana]     at HapiResponseAdapter.toError (/dev/shm/workspace/install/kibana-6/src/core/server/http/router/response_adapter.js:132:19)
[00:09:50]                   │ proc [kibana]     at HapiResponseAdapter.toHapiResponse (/dev/shm/workspace/install/kibana-6/src/core/server/http/router/response_adapter.js:86:19)
[00:09:50]                   │ proc [kibana]     at HapiResponseAdapter.handle (/dev/shm/workspace/install/kibana-6/src/core/server/http/router/response_adapter.js:81:17)
[00:09:50]                   │ proc [kibana]     at Router.handle (/dev/shm/workspace/install/kibana-6/src/core/server/http/router/router.js:160:34)
[00:09:50]                   │ proc [kibana]     at process._tickCallback (internal/process/next_tick.js:68:7)
[00:09:50]                   │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index template [metrics-system.socket_summary]
[00:09:50]                   │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index template [metrics-system.uptime]
[00:09:50]                   │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index template [metrics-system.users]
[00:09:50]                   │ info [o.e.x.i.a.TransportPutLifecycleAction] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index lifecycle policy [metrics-default]
[00:09:50]                   │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index template [logs-system.auth]
[00:09:50]                   │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index template [metrics-system.entropy]
[00:09:50]                   │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index template [metrics-system.filesystem]
[00:09:50]                   │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index template [metrics-system.cpu]
[00:09:50]                   │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index template [metrics-system.diskio]
[00:09:50]                   │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index template [metrics-system.load]
[00:09:50]                   │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index template [metrics-system.fsstat]
[00:09:50]                   │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index template [metrics-system.memory]
[00:09:51]                   │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index template [metrics-system.network_summary]
[00:09:51]                   │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index template [metrics-system.process_summary]
[00:09:51]                   │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index template [metrics-system.core]
[00:09:51]                   │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index template [metrics-system.process]
[00:09:51]                   │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index template [metrics-system.service]
[00:09:51]                   │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index template [metrics-system.raid]
[00:09:51]                   │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index template [metrics-system.network]
[00:09:51]                   │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index template [metrics-system.socket]
[00:09:51]                   │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index template [metrics-system.socket_summary]
[00:09:51]                   │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index template [logs-system.syslog]
[00:09:51]                   │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index template [metrics-system.uptime]
[00:09:51]                   │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index template [metrics-system.users]
[00:09:51]                   │ info [o.e.x.i.a.TransportPutLifecycleAction] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index lifecycle policy [logs-default]
[00:09:51]                   │ info [o.e.x.i.a.TransportPutLifecycleAction] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index lifecycle policy [events-default]
[00:09:51]                   │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding component template [logs-settings]
[00:09:51]                   │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding component template [events-settings]
[00:09:51]                   │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding component template [logs-mappings]
[00:09:51]                   │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding component template [events-mappings]
[00:09:51]                   │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding component template [metrics-mappings]
[00:09:51]                   │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding component template [metrics-settings]
[00:09:51]                   │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index template [events]
[00:09:51]                   │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index template [logs]
[00:09:51]                   │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xl-1591645358551354520] adding index template [metrics]
[00:09:52]                 └- ✖ fail: "apis Fleet Endpoints fleet_unenroll_agent allow to unenroll using a list of ids"
[00:09:52]                 │

[elasticmachine]

Pinging @elastic/endpoint-management (Team:Endpoint Management)

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[nchaulet]

The error seems to be because the package registry was not available

│ proc [kibana]   log   [20:44:26.765] [error][ingestManager][plugins] Error connecting to package registry at https://epr.elastic.co/epr/base/base-0.2.0.tar.gz: Error connecting to package registry at https://epr.elastic.co/epr/base/base-0.2.0.tar.gz: Backend unavailable, connection timeout

@spalger and @skh are working on having the package registry running in a docker container to avoid external call during tests.

[skh]

@spalger and @skh are working on having the package registry running in a docker container to avoid external call during tests.

See #68173

[jbudz]

#66579 reopened. i see 68173 is a draft and active. are we looking at that as the immediate fix or should we be looking for an intermediate solution?

[ruflin]

@jbudz Is the error consistent or was only happening once? I don't think we can rely on #66579 to unblock this even though I would like to see it in rather soonish.

[jbudz]

consistent, maybe 1 in 10 builds recently. there were global cloud outages yesterday which may be inflating that number. https://github.com/elastic/kibana/search?q=test+endpoint&type=Issues is a somewhat approximation

[ruflin]

@jbudz We uncovered yesterday an other issue where ES did implement a breaking change for us. Seeing data streams in this list made me wonder if different things contributed to the failures. But for the global cloud outages this could be a reason. Overall if it stays 1 in 10, this is way too much.

[nchaulet]

I will close this one in favor of #60865