Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Document data streams and custom index lifecycle policies #6553

Merged
merged 7 commits into from
Nov 22, 2021
Merged

Document data streams and custom index lifecycle policies #6553

merged 7 commits into from
Nov 22, 2021

Conversation

bmorelli25
Copy link
Member

@bmorelli25 bmorelli25 commented Nov 8, 2021 •
edited
Loading

Summary

  • Adds documentation for data streams
  • Adds documentation for default index lifecycle polices
  • Adds documentation for custom index lifecycle policies with component templates
  • Moves all "storage" content under the "Manage storage" heading

Preview

Manage storage bucket: https://apm-server_6553.docs-preview.app.elstc.co/guide/en/apm/guide/current/manage-storage.html
Data streams: https://apm-server_6553.docs-preview.app.elstc.co/guide/en/apm/guide/current/apm-data-streams.html
ILM/ILP: https://apm-server_6553.docs-preview.app.elstc.co/guide/en/apm/guide/current/ilm-how-to.html

Related

For elastic/observability-docs#1073
Closes elastic/observability-docs#1223
Closes elastic/observability-docs#1224

@bmorelli25 bmorelli25 added the Team:Docs Label for the Observability docs team label Nov 8, 2021
@bmorelli25 bmorelli25 requested a review from a team November 8, 2021 20:40
@bmorelli25 bmorelli25 self-assigned this Nov 8, 2021
@mergify mergify bot added the backport-skip Skip notification from the automated backport with mergify label Nov 8, 2021
@bmorelli25 bmorelli25 added backport-7.16 Automated backport with mergify to the 7.16 branch backport-8.0 Automated backport with mergify and removed backport-skip Skip notification from the automated backport with mergify labels Nov 8, 2021
@elastic elastic deleted a comment from mergify bot Nov 8, 2021
@bmorelli25 bmorelli25 mentioned this pull request Nov 8, 2021
Closed
25 tasks
@apmmachine
Copy link
Contributor

apmmachine commented Nov 8, 2021 •
edited
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2021-11-11T22:51:41.507+0000

  • Duration: 42 min 44 sec

  • Commit: 37170c9

Test stats 🧪

Test Results
Failed 0
Passed 6036
Skipped 19
Total 6055

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /hey-apm : Run the hey-apm benchmark.

  • /package : Generate and publish the docker images.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

axw
axw reviewed Nov 9, 2021
View reviewed changes
docs/data-streams.asciidoc Outdated Show resolved Hide resolved
docs/ilm-how-to.asciidoc Outdated
To apply your new index lifecycle policy to a data stream,
and to ensure the policy continues to be applied in future versions,
you must create a component template.
The component template name must start with `.`, follow the data stream naming scheme,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where did you get the . part from? AFAIK they do not have a dot prefix.

Copy link
Member Author

@bmorelli25 bmorelli25 Nov 9, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The dot (and these docs) are adapted from a tutorial that @hop-dev wrote. Mark, can you comment on whether the . is required or not?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@axw Thanks for pointing this out, I think this is a mistake in my original guide, you're right the component templates don't have a dot prefix.

@joshdover I can't see any record of us discussing having a dot prefix on the namespace specific component templates, I think we may be able to remove it? If so I'll get a PR in for the docs.

docs/ilm-how-to.asciidoc Outdated Show resolved Hide resolved
docs/how-to.asciidoc Show resolved Hide resolved
@bmorelli25
Copy link
Member Author

  • Removed . prefix
  • Updated the tutorial to edit the default <data-stream-name>@custom component template
  • Added a new section that briefly explains that it is possible to have more granular, namespace-level index lifecycle policies, and links to the tutorial in the Fleet User Guide for more information.

@bmorelli25
Copy link
Member Author

/test

axw
axw approved these changes Nov 15, 2021
View reviewed changes
Copy link
Member

@axw axw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great!

This was referenced Nov 15, 2021
Closed
Closed
@bmorelli25 bmorelli25 merged commit f76a51e into elastic:master Nov 22, 2021
@bmorelli25 bmorelli25 deleted the data-stream-and-other-docs branch November 22, 2021 22:34
mergify bot pushed a commit that referenced this pull request Nov 22, 2021
@bmorelli25
Document data streams and custom index lifecycle policies (#6553)
ad71be2 
(cherry picked from commit f76a51e)
@mergify mergify bot mentioned this pull request Nov 22, 2021
Merged
mergify bot pushed a commit that referenced this pull request Nov 22, 2021
@bmorelli25
Document data streams and custom index lifecycle policies (#6553)
c924a0d 
(cherry picked from commit f76a51e)
@mergify mergify bot mentioned this pull request Nov 22, 2021
Merged
bmorelli25 added a commit that referenced this pull request Nov 22, 2021
@mergify @bmorelli25
[7.16] Document data streams and custom index lifecycle policies (bac…
c2fa621 
…kport #6553) (#6689)

* Document data streams and custom index lifecycle policies (#6553)

(cherry picked from commit f76a51e)

* Update ilm-how-to.asciidoc

Co-authored-by: Brandon Morelli <brandon.morelli@elastic.co>
axw pushed a commit that referenced this pull request Nov 23, 2021
@mergify @bmorelli25
[8.0] Document data streams and custom index lifecycle policies (back…
58b3f12 
…port #6553) (#6688)

* Document data streams and custom index lifecycle policies (#6553)

(cherry picked from commit f76a51e)

* Update ilm-how-to.asciidoc

Co-authored-by: Brandon Morelli <brandon.morelli@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hop-dev hop-dev hop-dev left review comments

@axw axw axw approved these changes

Assignees

@bmorelli25 bmorelli25

Labels
backport-7.16 Automated backport with mergify to the 7.16 branch backport-8.0 Automated backport with mergify Team:Docs Label for the Observability docs team
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[APM] Document ILM policies and how to override them [APM] Document data streams
4 participants
@bmorelli25 @apmmachine @axw @hop-dev