[Fleet] Set default settings in component template instead of the index template #111197

nchaulet · 2021-09-03T18:21:33Z

Description

Why? To allow to override settings in template-component@custom component template settings need to be defined in a component template.

That PR change that and put the default settings in the template-component@settings component templates.

I did some refacto by moving this his own function and added some unit test we were missing here.

Known issue: this is going to be applied to newly installed package only and not to already installed package.

~~The index lifecycle policy is still defined in the index template? should we move it to the component template too?~~ Update all the index settings are now in the component template @settings.

How to test?

In a package:

you should be able to specify index template settings

In a custom component template:

You should be able to specify the settings index.number_of_replicas in a custom component template.

…ex template directly

elasticmachine · 2021-09-03T18:21:35Z

Pinging @elastic/fleet (Team:Fleet)

…3-set-default-settings-in-component-template

nchaulet · 2021-09-03T20:23:31Z

@elasticmachine merge upstream

…ponent-template

…3-set-default-settings-in-component-template

hop-dev · 2021-09-07T09:00:53Z

@nchaulet (cc @joshdover) Regarding the Index Lifecycle Policy, I have been looking at how users can customise data retention for integrations as part of elastic/observability-docs#986.

In the simplest case, where they wish to customise the retention of data for an integration for all namespaces, they could use the *@custom component template to set their own ILM policy, but I am not sure how realistic a use case this is and if we want to cater for it @joshdover?

If it isn't much trouble I think I would lean towards moving it to the component template

nchaulet · 2021-09-07T12:58:24Z

@hop-dev I think it make sense too. I just moved the ILM settings to the component template.

joshdover · 2021-09-07T13:18:51Z

In the simplest case, where they wish to customise the retention of data for an integration for all namespaces, they could use the *@custom component template to set their own ILM policy, but I am not sure how realistic a use case this is and if we want to cater for it @joshdover?

If it isn't much trouble I think I would lean towards moving it to the component template

Good catch @hop-dev, we need the custom component template to at least be able to set this for all namespaces. Thanks for making the change @nchaulet.

hop-dev · 2021-09-07T13:21:06Z

x-pack/plugins/fleet/server/services/epm/elasticsearch/template/default_settings.ts

+  ilmPolicy,
+  type,
+}: {
+  type: string;


Can type here only be logs or metrics? Should we express that in the type?

datastream.type is not typed to only logs and metrics, I think there is some other options too synthetics for example, I would keep this as string for now.

hop-dev · 2021-09-07T13:27:27Z

x-pack/plugins/fleet/server/services/epm/elasticsearch/template/default_settings.ts

+        },
+      },
+      // This is the default from Beats? So far seems to be a good value
+      refresh_interval: '5s',


Is now a good time to look at some of @ruflin 's recommendations here #104620 and trim some of these down? e.g potentially remove number_of_shards and refresh_interval

joshdover

Should we go ahead and incorporate the changes to the defaults that Adrien recommended here: #104620 (comment)

joshdover · 2021-09-07T13:27:50Z

x-pack/plugins/fleet/server/services/epm/elasticsearch/template/default_settings.ts

+      // What should be our default for the compression?
+      codec: 'best_compression',


@ruflin Do you know if we really should be using best_compression here? It appears Beats does not use this value and it does slow down field retrieval (though I don't know by how much):

The default value compresses stored data with LZ4 compression, but this can be set to best_compression which uses DEFLATE for a higher compression ratio, at the expense of slower stored fields performance. If you are updating the compression type, the new one will be applied after segments are merged. Segment merging can be forced using force merge.

@jpountz curious if you have any input on this as well

In my opinion, it is sensible to use best_speed by default for Enterprise Search data and best_compression for Observability and Security data.

x-pack/plugins/fleet/server/services/epm/elasticsearch/template/default_settings.test.ts

…e/default_settings.test.ts Co-authored-by: Josh Dover <1813008+joshdover@users.noreply.github.com>

nchaulet · 2021-09-07T14:00:09Z

@joshdover @hop-dev For #104620 I think the changes for the default value could come in a followup PR, it seems related but I like the benefits of having two PR/commits here it make things easier to follow, track future bugs. I will be happy to tackle #104620 later, if you want.

kibanamachine · 2021-09-07T16:08:37Z

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

💚 Build #151481 succeeded da8ae9d
💚 Build #151383 succeeded 69e151d
💔 Build #151359 failed 79044ee
💔 Build #151087 failed b40d3cf
💔 Build #151070 failed e352d7e

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @nchaulet

…ex template (elastic#111197)

kibanamachine · 2021-09-07T16:11:50Z

💚 Backport successful

Status	Branch	Result
✅	7.x

This backport PR will be merged automatically after passing CI.

…ex template (#111197) (#111419) Co-authored-by: Nicolas Chaulet <nicolas.chaulet@elastic.co>

…-link-to-kibana-app * 'master' of github.com:elastic/kibana: (61 commits) [Logs UI] Fix alert previews for thresholds of `0` (elastic#111150) [Archive Migration][Partial] discover apps-discover (elastic#110437) [APM] Set start date of APM ML job to -4 weeks (elastic#111375) [ML] APM Latency Correlations: Code consolidation. (elastic#110790) [Discover] Fix indices permission for multiline test (elastic#111284) [Detection Rules] Add 7.15 rules (elastic#111464) [Security Solution][Endpoint][Host Isolation] Hide isolate host option in alert details rather than disabling (elastic#111064) React version of angular license view (elastic#111317) [APM] Fix link in readme (elastic#111362) [Security Solution] add agent field to generator (elastic#111428) [Dashboard] Retain Tags on Quicksave (elastic#111015) Reorder App Search ingestion methods (elastic#111361) Port performance docs to new docs system. (elastic#111063) [Security Solution][RAC] Fixes updatedAt loading bug (elastic#111010) [sample data] update web log geo.src field to match country code of geo.coordinates (elastic#110885) [Security solution] [Endpoint] Fix bad artifact migration (elastic#111294) Fix copy typo. (elastic#111203) [build] Remove empty optimize directory (elastic#111393) [Maps] fix term join not updating when editing right field (elastic#111030) [Fleet] Set default settings in component template instead of the index template (elastic#111197) ... # Conflicts: # x-pack/plugins/reporting/public/management/__snapshots__/report_listing.test.tsx.snap # x-pack/plugins/reporting/public/management/report_listing.test.tsx

…ex template (elastic#111197)

[Fleet] Set default settings in component template instead of the ind…

1fb6fda

…ex template directly

nchaulet added v8.0.0 release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team auto-backport Deprecated - use backport:version if exact versions are needed v7.16.0 labels Sep 3, 2021

nchaulet self-assigned this Sep 3, 2021

nchaulet requested a review from a team as a code owner September 3, 2021 18:21

nchaulet added 2 commits September 3, 2021 14:21

Merge branch 'master' of github.com:elastic/kibana into feature-10560…

1365341

…3-set-default-settings-in-component-template

Fix tests

e352d7e

kibanamachine and others added 4 commits September 3, 2021 16:23

Merge branch 'master' into feature-105603-set-default-settings-in-com…

b40d3cf

…ponent-template

Merge branch 'master' of github.com:elastic/kibana into feature-10560…

218e702

…3-set-default-settings-in-component-template

Fix tests

79044ee

Fix integration tests

69e151d

hop-dev mentioned this pull request Sep 7, 2021

Document how to set data retention / ILM policies for integrations elastic/observability-docs#986

Closed

4 tasks

Add ilmPolicy definition to component template too

da8ae9d

joshdover self-requested a review September 7, 2021 13:19

hop-dev reviewed Sep 7, 2021

View reviewed changes

joshdover reviewed Sep 7, 2021

View reviewed changes

Update x-pack/plugins/fleet/server/services/epm/elasticsearch/templat…

42e3402

…e/default_settings.test.ts Co-authored-by: Josh Dover <1813008+joshdover@users.noreply.github.com>

hop-dev approved these changes Sep 7, 2021

View reviewed changes

nchaulet requested a review from joshdover September 7, 2021 14:43

joshdover approved these changes Sep 7, 2021

View reviewed changes

nchaulet merged commit b5f8d2e into elastic:master Sep 7, 2021

nchaulet deleted the feature-105603-set-default-settings-in-component-template branch September 7, 2021 16:09

kibanamachine mentioned this pull request Sep 7, 2021

[7.x] [Fleet] Set default settings in component template instead of the index template (#111197) #111419

Merged

kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Sep 7, 2021

[Fleet] Set default settings in component template instead of the ind…

62fa64f

…ex template (elastic#111197)

kibanamachine added a commit that referenced this pull request Sep 7, 2021

[Fleet] Set default settings in component template instead of the ind…

9025dea

…ex template (#111197) (#111419) Co-authored-by: Nicolas Chaulet <nicolas.chaulet@elastic.co>

chrisronline pushed a commit to chrisronline/kibana that referenced this pull request Sep 8, 2021

[Fleet] Set default settings in component template instead of the ind…

8f8fe30

…ex template (elastic#111197)

simitt mentioned this pull request Feb 3, 2022

[docs] Document how to customize index settings and ILM policies for managed apm-server elastic/apm-server#5988

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] Set default settings in component template instead of the index template #111197

[Fleet] Set default settings in component template instead of the index template #111197

nchaulet commented Sep 3, 2021 •

edited

Loading

elasticmachine commented Sep 3, 2021

nchaulet commented Sep 3, 2021

hop-dev commented Sep 7, 2021 •

edited

Loading

nchaulet commented Sep 7, 2021

joshdover commented Sep 7, 2021

hop-dev Sep 7, 2021

nchaulet Sep 7, 2021

hop-dev Sep 7, 2021

joshdover left a comment

joshdover Sep 7, 2021

joshdover Sep 7, 2021

jpountz Sep 7, 2021

nchaulet commented Sep 7, 2021

kibanamachine commented Sep 7, 2021

kibanamachine commented Sep 7, 2021

		// What should be our default for the compression?
		codec: 'best_compression',

[Fleet] Set default settings in component template instead of the index template #111197

[Fleet] Set default settings in component template instead of the index template #111197

Conversation

nchaulet commented Sep 3, 2021 • edited Loading

Description

How to test?

elasticmachine commented Sep 3, 2021

nchaulet commented Sep 3, 2021

hop-dev commented Sep 7, 2021 • edited Loading

nchaulet commented Sep 7, 2021

joshdover commented Sep 7, 2021

hop-dev Sep 7, 2021

Choose a reason for hiding this comment

nchaulet Sep 7, 2021

Choose a reason for hiding this comment

hop-dev Sep 7, 2021

Choose a reason for hiding this comment

joshdover left a comment

Choose a reason for hiding this comment

joshdover Sep 7, 2021

Choose a reason for hiding this comment

joshdover Sep 7, 2021

Choose a reason for hiding this comment

jpountz Sep 7, 2021

Choose a reason for hiding this comment

nchaulet commented Sep 7, 2021

kibanamachine commented Sep 7, 2021

💚 Build Succeeded

Metrics [docs]

History

kibanamachine commented Sep 7, 2021

💚 Backport successful

nchaulet commented Sep 3, 2021 •

edited

Loading

hop-dev commented Sep 7, 2021 •

edited

Loading