[Maps] Use default format when proxying EMS-files

[thomasneirynck]

NOTE: I added the api-test, just curious to see what it would do in CI. Need to discuss.

Summary

Closes #71975

Checklist

Delete any items that are not applicable to this PR.

• Unit or functional tests were updated or added to match the most common scenarios

For maintainers

• This was checked for breaking API changes and was labeled appropriately

[elasticmachine]

Pinging @elastic/kibana-gis (Team:Geo)

[thomasneirynck]

@elasticmachine merge upstream

[jsanz]

@thomasneirynck I've checked this branch but I can't start Kibana 😞

I've tested against master and 7.10 and both branches are working on my environment. I also tried to merge this branch against master and then even it run, I couldn't get into the login screen.

These are the errors I got on the console with a clean checkout of this PR branch:

Kibana log

...
  log   [13:24:36.335] [info][server] basepath proxy server running at http://0.0.0.0:5601/noh
server    log   [13:24:49.027] [info][plugins-service] Plugin "visTypeXy" is disabled.
server    log   [13:24:49.038] [info][plugins-service] Plugin "auditTrail" is disabled.
server    log   [13:24:49.579] [info][plugins-system] Setting up [96] plugins: [taskManager,licensing,globalSearch,globalSearchProviders,code,usageCollection,xpackLegacy,telemetryCollectionManager,telemetry,telemetryCollectionXpack,kibanaUsageCollection,securityOss,newsfeed,mapsLegacy,kibanaLegacy,translations,share,legacyExport,embeddable,uiActionsEnhanced,esUiShared,expressions,data,home,observability,cloud,console,consoleExtensions,apmOss,searchprofiler,painlessLab,grokdebugger,management,indexPatternManagement,advancedSettings,fileUpload,savedObjects,dashboard,visualizations,visTypeTable,visTypeTimelion,timelion,features,upgradeAssistant,security,snapshotRestore,enterpriseSearch,encryptedSavedObjects,ingestManager,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,dashboardMode,beatsManagement,transform,ingestPipelines,maps,licenseManagement,graph,dataEnhanced,visTypeVega,visTypeMarkdown,tileMap,regionMap,inputControlVis,visualize,bfetch,canvas,charts,visTypeMetric,lens,visTypeVislib,visTypeTimeseries,rollup,visTypeTagcloud,watcher,discover,discoverEnhanced,savedObjectsManagement,spaces,reporting,lists,eventLog,actions,case,alerts,stackAlerts,triggersActionsUi,ml,securitySolution,infra,monitoring,logstash,apm,uptime]
server    log   [13:24:50.147] [warning][config][plugins][security] Session cookies will be transmitted over insecure connections. This is not recommended.
server    log   [13:24:50.492] [info][config][plugins][reporting] Chromium sandbox provides an additional layer of protection, and is supported for Linux Ubuntu 20.04 OS. Automatically enabling Chromium sandbox.
server    log   [13:24:50.803] [info][monitoring][monitoring][plugins] config sourced from: production cluster
server    log   [13:24:51.287] [info][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...
server    log   [13:24:51.434] [info][savedobjects-service] Starting saved objects migrations
server    log   [13:24:51.559] [info][savedobjects-service] Creating index .kibana_1.
server    log   [13:24:51.605] [info][savedobjects-service] Creating index .kibana_task_manager_1.
server    log   [13:24:51.609] [error][data][elasticsearch] [mapper_parsing_exception]: Failed to parse mapping: unknown parameter [doc_values] on mapper [uiStateJSON] of type [text]
server    log   [13:24:51.611] [warning][savedobjects-service] Unable to connect to Elasticsearch. Error: mapper_parsing_exception
server    log   [13:24:51.612] [fatal][root] { ResponseError: mapper_parsing_exception
    at IncomingMessage.response.on (/home/j/src/elastic/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:257:25)
    at IncomingMessage.emit (events.js:203:15)
    at endReadableNT (_stream_readable.js:1145:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
  name: 'ResponseError',
  meta:
   { body: { error: [Object], status: 400 },
     statusCode: 400,
     headers:
      { 'content-type': 'application/json; charset=UTF-8',
        'content-length': '425' },
     meta:
      { context: null,
        request: [Object],
        name: 'elasticsearch-js',
        connection: [Object],
        attempts: 0,
        aborted: false } } }
server    log   [13:24:51.624] [info][plugins-system] Stopping all plugins.
server    log   [13:24:51.779] [info][savedobjects-service] Pointing alias .kibana_task_manager to .kibana_task_manager_1.
server    log   [13:24:51.873] [info][savedobjects-service] Finished in 268ms.

 FATAL  ResponseError: mapper_parsing_exception

 server crashed  with status code 1

Elasticsearch logs

...
   │ info [o.e.c.m.MetadataIndexTemplateService] [a] adding index template [synthetics] for index patterns [synthetics-*-*]
   │ info [o.e.c.r.a.AllocationService] [a] current.health="GREEN" message="Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.security-7][0]]])." previous.health="YELLOW" reason="shards started [[.security-7][0]]"
   │ info [o.e.x.i.a.TransportPutLifecycleAction] [a] adding index lifecycle policy [ml-size-based-ilm-policy]
   │ info [o.e.x.i.a.TransportPutLifecycleAction] [a] adding index lifecycle policy [metrics]
   │ info [o.e.x.i.a.TransportPutLifecycleAction] [a] adding index lifecycle policy [logs]
   │ info [o.e.x.i.a.TransportPutLifecycleAction] [a] adding index lifecycle policy [synthetics]
   │ info [o.e.x.i.a.TransportPutLifecycleAction] [a] adding index lifecycle policy [watch-history-ilm-policy]
   │ info [o.e.x.i.a.TransportPutLifecycleAction] [a] adding index lifecycle policy [ilm-history-ilm-policy]
   │ info [o.e.x.i.a.TransportPutLifecycleAction] [a] adding index lifecycle policy [slm-history-ilm-policy]
   │ info [o.e.l.LicenseService] [a] license [3c7f1e2f-274c-40ab-81a6-383b3caa5fc8] mode [basic] - valid
   │ info [o.e.x.s.s.SecurityStatusChangeListener] [a] Active license is now [BASIC]; Security is enabled
   │ info [o.e.c.m.MetadataCreateIndexService] [a] failed on parsing mappings on index creation [.kibana_1]
   │      org.elasticsearch.index.mapper.MapperParsingException: Failed to parse mapping: unknown parameter [doc_values] on mapper [uiStateJSON] of type [text]
   │      	at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:289) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:262) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.updateIndexMappingsAndBuildSortOrder(MetadataCreateIndexService.java:918) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.lambda$applyCreateIndexWithTemporaryService$2(MetadataCreateIndexService.java:412) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.indices.IndicesService.withTempIndexService(IndicesService.java:621) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexWithTemporaryService(MetadataCreateIndexService.java:410) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequestWithV1Templates(MetadataCreateIndexService.java:488) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:373) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:380) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.cluster.metadata.MetadataCreateIndexService$1.execute(MetadataCreateIndexService.java:303) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:697) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:319) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:214) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:674) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
   │      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
   │      	at java.lang.Thread.run(Thread.java:832) [?:?]
   │      Caused by: org.elasticsearch.index.mapper.MapperParsingException: unknown parameter [doc_values] on mapper [uiStateJSON] of type [text]
   │      	at org.elasticsearch.index.mapper.ParametrizedFieldMapper$Builder.parse(ParametrizedFieldMapper.java:621) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.index.mapper.ParametrizedFieldMapper$TypeParser.parse(ParametrizedFieldMapper.java:666) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.index.mapper.ParametrizedFieldMapper$TypeParser.parse(ParametrizedFieldMapper.java:651) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:320) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:237) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parse(ObjectMapper.java:209) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:320) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:237) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.index.mapper.RootObjectMapper$TypeParser.parse(RootObjectMapper.java:138) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:111) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:99) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:287) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │      	... 22 more
   │ info [o.e.c.m.MetadataCreateIndexService] [a] [.kibana_task_manager_1] creating index, cause [api], templates [], shards [1]/[1]
   │ info [o.e.c.r.a.AllocationService] [a] updating number_of_replicas to [0] for indices [.kibana_task_manager_1]
   │ info [o.e.c.r.a.AllocationService] [a] current.health="GREEN" message="Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.kibana_task_manager_1][0]]])." previous.health="YELLOW" reason="shards started [[.kibana_task_manager_1][0]]"

[thomasneirynck]

@jsanz I'm seeing this on other PRs as well (e.g. #79934) tbd..

[thomasneirynck]

@elasticmachine merge upstream

[jsanz]

I could test this locally with:

• Serve locally a "broken" manifest using the World Countries new data from GeoJSON datasets ems-file-service#192 and putting the GeoJSON format before the default TopoJSON
• Tested that this set up is indeed broken in branch 7.10, where the GeoJSON is pulled but expecting a TopoJSON.
• Tested that in this branch the correct dataset is pulled and correctly processed and presented

[thomasneirynck]

@elasticmachine merge upstream

[nickpeihl]

just a couple of nits in tests. but otherwise lgtm. tested proxy with multiple formats and confirmed it works.

wrt testing. rather than use the cloud ems service, I wonder if we shouldn't create a local EMS mock plugin similar to how the newsfeed is mocked for tests? maybe out of scope for this PR though.

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: d0740e2

Metrics [docs]

✅ unchanged

History

• 💚 Build #81111 succeeded 5cd0147
• 💚 Build #80730 succeeded 5d68ad0
• 💚 Build #80467 succeeded 5664f3c
• 💚 Build #80348 succeeded 9a47be6
• 💛 Build #80298 was flaky 6e7a9ad

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream