[Ingest Manager] Set _meta in the index.mappings

[afharo]

Summary

Add the _meta payload to the template's mappings. This way it is copied over to the indices created based on that template (under mappings._meta).

This way, GET {DATASTREAM_NAME}/_mapping?filter_path=*.mappings._meta will return the indices created based of the data-stream and it will contain Ingest Manager's custom _meta for the UI to identify them.

Related to #71419 and #59195.

Checklist

Delete any items that are not applicable to this PR.

• Unit or functional tests were updated or added to match the most common scenarios

For maintainers

• This was checked for breaking API changes and was labeled appropriately

[elasticmachine]

Pinging @elastic/ingest-management (Feature:EPM)

[afharo]

@elasticmachine merge upstream

[kibanamachine]

💛 Build succeeded, but was flaky

• continuous-integration/kibana-ci/pull-request
• Commit: a7f055f
• Flaky suites:
  • xpack-kibana-ciGroup7

---

Test Failures

Kibana Pipeline / kibana-xpack-agent / X-Pack Endpoint Functional Tests.x-pack/test/security_solution_endpoint/apps/endpoint.endpoint "before all" hook in "endpoint"

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

[00:00:00]       │
[00:00:00]         └-: endpoint
[00:00:00]           └-> "before all" hook
[00:00:00]           └-> "before all" hook
[00:00:00]             │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xl-1595240833054335465] [logs-index_pattern_placeholder] creating index, cause [api], templates [], shards [1]/[1]
[00:00:00]             │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xl-1595240833054335465] [metrics-index_pattern_placeholder] creating index, cause [api], templates [], shards [1]/[1]
[00:00:00]             │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xl-1595240833054335465] [events-index_pattern_placeholder] creating index, cause [api], templates [], shards [1]/[1]
[00:00:00]             │ proc [kibana]  error  [11:46:57.414] [warning][process] UnhandledPromiseRejectionWarning: Error: Cannot install or update to an out-of-date package
[00:00:00]             │ proc [kibana]     at installPackage (/dev/shm/workspace/install/kibana-8/x-pack/plugins/ingest_manager/server/services/epm/packages/install.js:127:82)
[00:00:00]             │ proc [kibana]     at process._tickCallback (internal/process/next_tick.js:68:7)
[00:00:00]             │ proc [kibana]     at emitWarning (internal/process/promises.js:97:15)
[00:00:00]             │ proc [kibana]     at emitPromiseRejectionWarnings (internal/process/promises.js:143:7)
[00:00:00]             │ proc [kibana]     at process._tickCallback (internal/process/next_tick.js:69:34)
[00:00:00]             │ proc [kibana]  error  [11:46:57.415] [warning][process] Error: Cannot install or update to an out-of-date package
[00:00:00]             │ proc [kibana]     at installPackage (/dev/shm/workspace/install/kibana-8/x-pack/plugins/ingest_manager/server/services/epm/packages/install.js:127:82)
[00:00:00]             │ proc [kibana]     at process._tickCallback (internal/process/next_tick.js:68:7)
[00:00:00]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [kibana-ci-immutable-ubuntu-16-tests-xl-1595240833054335465] added role [fleet_enroll]
[00:00:00]             │ info [o.e.x.s.a.u.TransportPutUserAction] [kibana-ci-immutable-ubuntu-16-tests-xl-1595240833054335465] added user [fleet_enroll]
[00:00:00]             │ info Taking screenshot "/dev/shm/workspace/kibana/x-pack/test/functional/screenshots/failure/endpoint _before all_ hook.png"
[00:00:00]             │ proc [kibana]  error  [11:46:57.291]  Error: Internal Server Error
[00:00:00]             │ proc [kibana]     at HapiResponseAdapter.toError (/dev/shm/workspace/install/kibana-8/src/core/server/http/router/response_adapter.js:132:19)
[00:00:00]             │ proc [kibana]     at HapiResponseAdapter.toHapiResponse (/dev/shm/workspace/install/kibana-8/src/core/server/http/router/response_adapter.js:86:19)
[00:00:00]             │ proc [kibana]     at HapiResponseAdapter.handle (/dev/shm/workspace/install/kibana-8/src/core/server/http/router/response_adapter.js:81:17)
[00:00:00]             │ proc [kibana]     at Router.handle (/dev/shm/workspace/install/kibana-8/src/core/server/http/router/router.js:162:34)
[00:00:00]             │ proc [kibana]     at process._tickCallback (internal/process/next_tick.js:68:7)
[00:00:00]             │ info Current URL is: data:/,
[00:00:00]             │ info Saving page source to: /dev/shm/workspace/kibana/x-pack/test/functional/failure_debug/html/endpoint _before all_ hook.html
[00:00:00]             └- ✖ fail: endpoint "before all" hook in "endpoint"
[00:00:00]             │      Error: expected 200 "OK", got 500 "Internal Server Error"
[00:00:00]             │       at Test._assertStatus (/dev/shm/workspace/kibana/node_modules/supertest/lib/test.js:268:12)
[00:00:00]             │       at Test._assertFunction (/dev/shm/workspace/kibana/node_modules/supertest/lib/test.js:283:11)
[00:00:00]             │       at Test.assert (/dev/shm/workspace/kibana/node_modules/supertest/lib/test.js:173:18)
[00:00:00]             │       at assert (/dev/shm/workspace/kibana/node_modules/supertest/lib/test.js:131:12)
[00:00:00]             │       at /dev/shm/workspace/kibana/node_modules/supertest/lib/test.js:128:5
[00:00:00]             │       at Test.Request.callback (/dev/shm/workspace/kibana/node_modules/superagent/lib/node/index.js:718:3)
[00:00:00]             │       at parser (/dev/shm/workspace/kibana/node_modules/superagent/lib/node/index.js:906:18)
[00:00:00]             │       at IncomingMessage.res.on (/dev/shm/workspace/kibana/node_modules/superagent/lib/node/parsers/json.js:19:7)
[00:00:00]             │       at endReadableNT (_stream_readable.js:1145:12)
[00:00:00]             │       at process._tickCallback (internal/process/next_tick.js:63:19)
[00:00:00]             │ 
[00:00:00]             │ 

Stack Trace

Error: expected 200 "OK", got 500 "Internal Server Error"
    at Test._assertStatus (/dev/shm/workspace/kibana/node_modules/supertest/lib/test.js:268:12)
    at Test._assertFunction (/dev/shm/workspace/kibana/node_modules/supertest/lib/test.js:283:11)
    at Test.assert (/dev/shm/workspace/kibana/node_modules/supertest/lib/test.js:173:18)
    at assert (/dev/shm/workspace/kibana/node_modules/supertest/lib/test.js:131:12)
    at /dev/shm/workspace/kibana/node_modules/supertest/lib/test.js:128:5
    at Test.Request.callback (/dev/shm/workspace/kibana/node_modules/superagent/lib/node/index.js:718:3)
    at parser (/dev/shm/workspace/kibana/node_modules/superagent/lib/node/index.js:906:18)
    at IncomingMessage.res.on (/dev/shm/workspace/kibana/node_modules/superagent/lib/node/parsers/json.js:19:7)
    at endReadableNT (_stream_readable.js:1145:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)

---

Build metrics

✅ unchanged

History

• 💛 Build #62204 was flaky ff4155b

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream