[Fleet] Remove ILM policy from fleet files indices and make them hidden #94651

hop-dev · 2023-03-22T21:07:37Z

There is a problem with the way we set up the indices that these ILM policies affect, causing the error below, it's too late to implement a full fix in 8.7.0 so for now we are removing the policy from the file upload indices.

In investgating the issue I also noticed that the index templates do not hide the indices so I have added hidden: true also.

Testing steps

To check that diagnostic upload is still working:

run elasticsearch locally using gradlew run
generate a sevrice token (probably a better way to do this but this is how I do it:

curl -XPOST -u elastic:password http://localhost:9200/_security/service/elastic/kibana/credential/token/token1

add the service token to kibana config:

elasticsearch.serviceAccountToken: "token"
elasticsearch.hosts: ["http://localhost:9200"]

Add an 8.7.0 fleet server and agent to fleet
request a diagnostic file, it should be uploaded successfully

elasticsearchmachine · 2023-03-22T21:11:48Z

Pinging @elastic/es-core-infra (Team:Core/Infra)

…53499) ## Summary We are having to remove the ILM policies that delete data after 30 days from 8.7.0 in elastic/elasticsearch#94651, so removing this text as well: "Fleet will automatically remove old diagnostics files after 30 days." How they look now: <img width="769" alt="Screenshot 2023-03-22 at 20 57 40" src="https://user-images.githubusercontent.com/3315046/227036926-29a12823-002f-483d-8ead-5bf759bdb7d4.png"> <img width="1247" alt="Screenshot 2023-03-22 at 20 57 33" src="https://user-images.githubusercontent.com/3315046/227036928-1dd4ae35-8d9d-4630-a769-33b943d19269.png">

…astic#153499) ## Summary We are having to remove the ILM policies that delete data after 30 days from 8.7.0 in elastic/elasticsearch#94651, so removing this text as well: "Fleet will automatically remove old diagnostics files after 30 days." How they look now: <img width="769" alt="Screenshot 2023-03-22 at 20 57 40" src="https://user-images.githubusercontent.com/3315046/227036926-29a12823-002f-483d-8ead-5bf759bdb7d4.png"> <img width="1247" alt="Screenshot 2023-03-22 at 20 57 33" src="https://user-images.githubusercontent.com/3315046/227036928-1dd4ae35-8d9d-4630-a769-33b943d19269.png"> (cherry picked from commit a3fa269)

…ys (#153499) (#153502) # Backport This will backport the following commits from `main` to `8.7`: - [[Fleet] Remove text saying file uploads are deleted after 30 days (#153499)](#153499)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  Co-authored-by: Mark Hopkin <mark.hopkin@elastic.co>

williamrandolph

If this code had already gotten out into production, you would need some kind of migration to fix up existing indices created from these templates, but since the affected code isn't on the 8.6 branch, this won't be a problem.

Should we remove the ILM policies .fleet-file-data-ilm-policy and .fleet-files-ilm-policy as well? Or would they be unchanged in the full fix?

hop-dev · 2023-03-23T06:48:05Z

@williamrandolph we will need the ILM policies for the full fix, if removing them prevents us from shipping that in 8.7.x I'd rather keep them there if possible

andreidan

Thanks for working on this Mark

I left a few comments.

andreidan · 2023-03-23T10:46:18Z

x-pack/plugin/core/src/main/resources/fleet-files.json

@@ -10,8 +10,8 @@
  },
  "template": {
    "settings": {
-      "index.lifecycle.name": ".fleet-files-ilm-policy",


Is the fleet-file-data-ilm-policy.json policy still needed?

we will need the ILM policies for the full fix, if removing them prevents us from shipping that in 8.7.x I'd rather keep them there if possible?

Can you expand a bit on what the full fix is?

Looks like the plan is here: elastic/kibana#153504

In order to set the setting on all rolled over indices, we will have to have an index template per integration. E,g instead of .fleet-files-* we will have to have one for .fleet-files-endpoint-* and .fleet-files-agent-*. This could be created dynamically by fleet, we already have the logic for creating the backing indices at integration install time and fleet start time, see here.

We can still use the ILM policies hard coded into elasticsearch.

Sure, its not 100% as we only discovered this yesterday, but we will likely be looking to have kibana create the index templates at package install time, which will then refer to these ILM policies. For now removing ILM from the current index template means the feature still works.

Our issue at the moment is that we were trying to use one index template for all of our integrations that will be using this feature, but as each integration has a different alias, we aren't able to set the ilm rollover_alias on the index template, we thought setting it on the first write index works but we were wrong.

Moving to kibana creating the index templates will mean we wont have to have an index template for each integration in elasticsearch repo.

I have an issue for the long term fix here elastic/kibana#153504

andreidan · 2023-03-23T10:50:51Z

x-pack/plugin/core/src/main/resources/fleet-files.json

@@ -10,8 +10,8 @@
  },
  "template": {
    "settings": {
-      "index.lifecycle.name": ".fleet-files-ilm-policy",
-      "index.auto_expand_replicas": "0-1"
+      "index.auto_expand_replicas": "0-1",


I think this will not have effect for deployments updating to the version where this change is released.
See https://github.com/elastic/elasticsearch/blob/main/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/template/IndexTemplateRegistry.java#L296

You'll have to bump the template version in FleetTemplateRegistry https://github.com/elastic/elasticsearch/blob/main/x-pack/plugin/fleet/src/main/java/org/elasticsearch/xpack/fleet/FleetTemplateRegistry.java#L27
and please also document the change next to the new version
e.g.

elasticsearch/x-pack/plugin/ilm/src/main/java/org/elasticsearch/xpack/ilm/history/ILMHistoryTemplateRegistry.java

Line 38 in 3692354

// version 5: convert to data stream

Please also test the branch as a standalone elasticsearch run and upgrade from a previous version to this branch to see that the templates and your code behaves as expected.
You can build a tarball using ./gradlew clean :distribution:archives:darwin-tar:assemble (change distribution accordingly) or run elasticsearch using ./gradlew run

These templates were added in 8.7 so nobody should have these changes yet, so is this required?

Ah gotcha, then we should be fine here

My understanding is that there are no such deployments, since these templates are being newly introduced in 8.7: #91413

That PR seems so distant, I can't believe it's targeting 8.7 and not an older version :)

Ha it sat blocked on another team's effort for about a month if I recall. This file upload feature was slotted for 8.6 initially but was eventually punted, with a bulk of the work landing early on in the 8.7 development cycle.

andreidan

LGTM, thanks Mark

elasticsearchmachine · 2023-03-23T11:25:14Z

💚 Backport successful

Status	Branch	Result
✅	8.7

…en (elastic#94651) * Remove ILM policy from files indices and make them hidden * use correct setting for hidden and auto_expand * fix tests * remove unused imports

…en (#94651) (#94668) * Remove ILM policy from files indices and make them hidden * use correct setting for hidden and auto_expand * fix tests * remove unused imports

Remove ILM policy from files indices and make them hidden

9b2975a

hop-dev added auto-backport Automatically create backport pull requests when merged v8.7.0 labels Mar 22, 2023

hop-dev self-assigned this Mar 22, 2023

elasticsearchmachine added v8.8.0 needs:triage Requires assignment of a team area label external-contributor Pull request authored by a developer outside the Elasticsearch team labels Mar 22, 2023

hop-dev added the >non-issue label Mar 22, 2023

hop-dev mentioned this pull request Mar 22, 2023

[Fleet] Remove text saying file uploads are deleted after 30 days elastic/kibana#153499

Merged

jen-huang requested a review from kpollich March 22, 2023 21:10

hop-dev added the Team:Fleet label Mar 22, 2023

elasticsearchmachine removed the needs:triage Requires assignment of a team area label label Mar 22, 2023

hop-dev added :Core/Infra/Plugins Plugin API and infrastructure Team:Core/Infra Meta label for core/infra team labels Mar 22, 2023

hop-dev added 2 commits March 22, 2023 21:15

use correct setting for hidden and auto_expand

a7687e9

fix tests

230f7d2

remove unused imports

3f41345

williamrandolph reviewed Mar 23, 2023

View reviewed changes

andreidan requested changes Mar 23, 2023

View reviewed changes

andreidan approved these changes Mar 23, 2023

View reviewed changes

hop-dev merged commit 6d6cbb0 into elastic:main Mar 23, 2023

hop-dev deleted the 94647-do-not-rollover-fleet-files-indices branch March 23, 2023 11:23

hop-dev mentioned this pull request Mar 23, 2023

[8.7] [Fleet] Remove ILM policy from fleet files indices and make them hidden (#94651) #94668

Merged

jen-huang mentioned this pull request Apr 21, 2023

[Fleet]: Agent diagnostics failed for Hosted fleet server under Diagnostics tab. elastic/kibana#155483

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] Remove ILM policy from fleet files indices and make them hidden #94651

[Fleet] Remove ILM policy from fleet files indices and make them hidden #94651

hop-dev commented Mar 22, 2023

elasticsearchmachine commented Mar 22, 2023

williamrandolph left a comment

hop-dev commented Mar 23, 2023

andreidan left a comment

andreidan Mar 23, 2023

hop-dev Mar 23, 2023

andreidan Mar 23, 2023

williamrandolph Mar 23, 2023 •

edited

Loading

hop-dev Mar 23, 2023

andreidan Mar 23, 2023 •

edited

Loading

hop-dev Mar 23, 2023

andreidan Mar 23, 2023

williamrandolph Mar 23, 2023

andreidan Mar 23, 2023

kpollich Mar 23, 2023

andreidan left a comment

elasticsearchmachine commented Mar 23, 2023

[Fleet] Remove ILM policy from fleet files indices and make them hidden #94651

[Fleet] Remove ILM policy from fleet files indices and make them hidden #94651

Conversation

hop-dev commented Mar 22, 2023

Testing steps

elasticsearchmachine commented Mar 22, 2023

williamrandolph left a comment

Choose a reason for hiding this comment

hop-dev commented Mar 23, 2023

andreidan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

williamrandolph Mar 23, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andreidan Mar 23, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andreidan left a comment

Choose a reason for hiding this comment

elasticsearchmachine commented Mar 23, 2023

💚 Backport successful

williamrandolph Mar 23, 2023 •

edited

Loading

andreidan Mar 23, 2023 •

edited

Loading