Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make use of lazy rollovers for invoked unconditional rollover #174480

Closed
dakrone opened this issue Jan 8, 2024 · 5 comments · Fixed by #174790 or #176565
Closed

Make use of lazy rollovers for invoked unconditional rollover #174480

dakrone opened this issue Jan 8, 2024 · 5 comments · Fixed by #174790 or #176565
Assignees
@nchaulet
Labels
Feature:EPM Fleet team's Elastic Package Manager (aka Integrations) project Team:Fleet Team label for Observability Data Collection Fleet team

Comments

@dakrone
Copy link
Member

dakrone commented Jan 8, 2024 •
edited
Loading

Describe the feature:
Fleet/EPM unconditionally force a rollover inside of Elasticsearch when mappings for an integration are updated. We've recently introduced the ?lazy parameter inside of ES to allow rolling over a data stream on next document write instead of immediately. This would allow us to avoid a situation where Fleet rolls over the same index multiple times despite being completely empty (and avoid extra empty indices). It'd be great if these rollover invocations can use this flag.

Related ES PR: elastic/elasticsearch#103309
@dakrone dakrone added Feature:EPM Fleet team's Elastic Package Manager (aka Integrations) project Team:Fleet Team label for Observability Data Collection Fleet team labels Jan 8, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/fleet (Feature:EPM)

@elasticmachine
Copy link
Contributor

Pinging @elastic/fleet (Team:Fleet)

@kpollich
Copy link
Member

kpollich commented Jan 8, 2024

I think this would be a very small change with pretty high impact, so I'm prioritizing this highly in our next sprint.

@nchaulet nchaulet self-assigned this Jan 12, 2024
@nchaulet nchaulet mentioned this issue Jan 12, 2024
Merged
@jlind23 jlind23 reopened this Jan 25, 2024
@jlind23
Copy link
Contributor

jlind23 commented Jan 25, 2024

Reopening this as we had to revert the PR due an issue on the ES side: elastic/elasticsearch#104732

@jlind23
Copy link
Contributor

jlind23 commented Feb 9, 2024 •
edited
Loading

@nchaulet the related ES bug has been fixed so feel free to merge this again 8.13.

@juliaElastic juliaElastic mentioned this issue Feb 9, 2024
Merged
juliaElastic added a commit that referenced this issue Feb 9, 2024
@juliaElastic @nchaulet
[Fleet] Make datastream rollover lazy (#174790) (#176565)
240e5ef 
## Summary

Add back changes in #174790 after
elastic/elasticsearch#104732 is fixed

Resolve #174480

Co-authored-by: Nicolas Chaulet <nicolas.chaulet@elastic.co>
CoenWarmer pushed a commit to CoenWarmer/kibana that referenced this issue Feb 15, 2024
@juliaElastic @nchaulet @CoenWarmer
[Fleet] Make datastream rollover lazy (elastic#174790) (elastic#176565)
c7655ce 
## Summary

Add back changes in elastic#174790 after
elastic/elasticsearch#104732 is fixed

Resolve elastic#174480

Co-authored-by: Nicolas Chaulet <nicolas.chaulet@elastic.co>
fkanout pushed a commit to fkanout/kibana that referenced this issue Mar 4, 2024
@juliaElastic @nchaulet @fkanout
[Fleet] Make datastream rollover lazy (elastic#174790) (elastic#176565)
336ad2b 
## Summary

Add back changes in elastic#174790 after
elastic/elasticsearch#104732 is fixed

Resolve elastic#174480

Co-authored-by: Nicolas Chaulet <nicolas.chaulet@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nchaulet nchaulet

Labels
Feature:EPM Fleet team's Elastic Package Manager (aka Integrations) project Team:Fleet Team label for Observability Data Collection Fleet team
Projects
None yet
Milestone
No milestone
5 participants
@dakrone @nchaulet @kpollich @elasticmachine @jlind23