Introduce lazy rollover for mapping updates in data streams #103309

gmarouli · 2023-12-12T09:35:49Z

Currently, we recommend an unconditional rollover to apply mapping changes to data streams. However, as described in #89346, this can result in many empty backing indices.

In this PR we implement the idea to introduce a flag, that a data stream needs to be rolloved over before the next document is indexed.

The work is split into the following components:

Introduce a flag for lazy rollover in the rollover request (part of this PR).
Introduce a flag in the data stream state to mark that this data streams needs to be rolled over next time a document gets indexed (part of this PR).
During indexing a document, if the data stream has the flag, rollover before indexing completes. (Indicative PR: Introduce lazy rollover for mapping updates in data streams [part 2] gmarouli/elasticsearch#6)
Update documentation Introduce lazy rollover for mapping updates in data streams [part 2] gmarouli/elasticsearch#6

Relates to #89346

elasticsearchmachine · 2023-12-12T09:36:41Z

Pinging @elastic/es-data-management (Team:Data Management)

elasticsearchmachine · 2023-12-12T09:37:09Z

Hi @gmarouli, I've created a changelog YAML for you.

gmarouli · 2023-12-13T10:03:39Z

@elasticmachine update branch

elasticsearchmachine · 2023-12-13T11:02:10Z

Hi @gmarouli, I've updated the changelog YAML for you.

This reverts commit b145caa.

gmarouli · 2023-12-13T15:23:58Z

server/src/main/java/org/elasticsearch/cluster/metadata/DataStream.java

+            DataStream.isFailureStoreEnabled() && args[11] != null && (boolean) args[11],
+            DataStream.isFailureStoreEnabled() && args[12] != null ? (List<Index>) args[12] : List.of(),


Fields behind feature flags need to be parsed last to ensure that the parser will not fail when the feature flag is disabled.

Maybe we should add that as a comment, since once the feature flags are removed it'll just look weird and out of order

Or potentially we could pull them out into temporary variables so the order can be changed at whim (either way it's a super minor thing, so no pressure)

Let me know if I got correctly what you had in mind :).

elasticsearchmachine · 2023-12-13T15:26:19Z

Hi @gmarouli, I've updated the changelog YAML for you.

dakrone

I think this looks great so far! I left a few minor comments

docs/reference/indices/rollover-index.asciidoc

...r/src/main/java/org/elasticsearch/action/admin/indices/rollover/TransportRolloverAction.java

server/src/main/java/org/elasticsearch/action/datastreams/GetDataStreamAction.java

dakrone · 2023-12-13T15:59:37Z

server/src/main/java/org/elasticsearch/cluster/metadata/DataStream.java

+            DataStream.isFailureStoreEnabled() && args[11] != null && (boolean) args[11],
+            DataStream.isFailureStoreEnabled() && args[12] != null ? (List<Index>) args[12] : List.of(),


Maybe we should add that as a comment, since once the feature flags are removed it'll just look weird and out of order

dakrone · 2023-12-13T16:02:21Z

server/src/main/java/org/elasticsearch/cluster/metadata/DataStream.java

+            DataStream.isFailureStoreEnabled() && args[11] != null && (boolean) args[11],
+            DataStream.isFailureStoreEnabled() && args[12] != null ? (List<Index>) args[12] : List.of(),


Or potentially we could pull them out into temporary variables so the order can be changed at whim (either way it's a super minor thing, so no pressure)

dakrone

So I was playing around with this, and it appears that I can crash ES with the following:

$ ./gradlew run

Then:

POST /logs-bar-bag/_doc
{"@timestamp": "2023-06-01", "message": "my message"}

POST /logs-bar-bag/_rollover?dry_run&lazy
{
  "conditions": {
    "max_docs": 1
  }
}

Which then crashes with:

[2023-12-21T11:29:02,769][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [runTask-0] fatal error in thread [elasticsearch[runTask-0][management][T#3]], exiting java.lang.AssertionError: org.elasticsearch.ElasticsearchException: org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener/org.elasticsearch.action.support.ContextPreservingActionListener/org.elasticsearch.tasks.TaskManager$1{org.elasticsearch.rest.action.RestCancellableNodeClient$1@69e12f1c}{Task{id=219, type='transport', action='indices:admin/rollover', description='', parentTask=unset, startTime=1703183342758, startTimeNanos=106715515529625}}/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter$$Lambda$7006/0x0000000801c2ef98@7b5bf01d/org.elasticsearch.action.support.master.TransportMasterNodeAction$$Lambda$7208/0x0000000801ca50a0@7e331720
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListener$3.assertFirstRun(ActionListener.java:305)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:310)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onResponse(ActionListenerImplementations.java:182)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.admin.indices.rollover.TransportRolloverAction.lambda$masterOperation$1(TransportRolloverAction.java:215)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:312)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:202)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:196)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:95)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:308)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:228)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:131)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:107)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.support.CancellableFanOut$SubtasksCompletionHandler.run(CancellableFanOut.java:201)
	at org.elasticsearch.base@8.13.0-SNAPSHOT/org.elasticsearch.core.AbstractRefCounted$1.closeInternal(AbstractRefCounted.java:118)
	at org.elasticsearch.base@8.13.0-SNAPSHOT/org.elasticsearch.core.AbstractRefCounted.decRef(AbstractRefCounted.java:70)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.support.RefCountingRunnable.close(RefCountingRunnable.java:112)
	at org.elasticsearch.base@8.13.0-SNAPSHOT/org.elasticsearch.core.Releasables$3.close(Releasables.java:142)
	at org.elasticsearch.base@8.13.0-SNAPSHOT/org.elasticsearch.core.IOUtils.close(IOUtils.java:71)
	at org.elasticsearch.base@8.13.0-SNAPSHOT/org.elasticsearch.core.Releasables.close(Releasables.java:33)
	at org.elasticsearch.base@8.13.0-SNAPSHOT/org.elasticsearch.core.Releasables.closeExpectNoException(Releasables.java:57)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$2.run(ActionListenerImplementations.java:49)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$RunAfterActionListener.onResponse(ActionListenerImplementations.java:262)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:312)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:49)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.transport.TransportService$UnregisterChildTransportResponseHandler.handleResponse(TransportService.java:1709)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1425)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1524)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.transport.TransportService$DirectResponseChannel$1.doRun(TransportService.java:1501)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.elasticsearch.ElasticsearchException: org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener/org.elasticsearch.action.support.ContextPreservingActionListener/org.elasticsearch.tasks.TaskManager$1{org.elasticsearch.rest.action.RestCancellableNodeClient$1@69e12f1c}{Task{id=219, type='transport', action='indices:admin/rollover', description='', parentTask=unset, startTime=1703183342758, startTimeNanos=106715515529625}}/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter$$Lambda$7006/0x0000000801c2ef98@7b5bf01d/org.elasticsearch.action.support.master.TransportMasterNodeAction$$Lambda$7208/0x0000000801ca50a0@7e331720
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListener$3.assertFirstRun(ActionListener.java:304)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListener$3.onFailure(ActionListener.java:321)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.lambda$doStart$2(TransportMasterNodeAction.java:233)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.acceptException(ActionListenerImplementations.java:186)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onFailure(ActionListenerImplementations.java:191)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.admin.indices.rollover.TransportRolloverAction.masterOperation(TransportRolloverAction.java:155)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.admin.indices.rollover.TransportRolloverAction.masterOperation(TransportRolloverAction.java:65)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.support.master.TransportMasterNodeAction.executeMasterOperation(TransportMasterNodeAction.java:125)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.lambda$doStart$3(TransportMasterNodeAction.java:236)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:95)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:238)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:236)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:173)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:55)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:87)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:165)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:212)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:622)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:21)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:626)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:622)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:85)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:626)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:622)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:98)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:626)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:622)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.interceptor.ShardSearchRequestInterceptor.intercept(ShardSearchRequestInterceptor.java:24)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:626)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:622)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:124)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:626)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:622)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.interceptor.DlsFlsLicenseRequestInterceptor.intercept(DlsFlsLicenseRequestInterceptor.java:106)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:626)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:622)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:27)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:626)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:622)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.interceptor.SearchRequestCacheDisablingInterceptor.intercept(SearchRequestCacheDisablingInterceptor.java:53)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:622)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:607)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$13(AuthorizationService.java:505)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:1033)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:999)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:401)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:308)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:228)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:131)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:107)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:1079)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:381)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:498)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:435)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$3(AuthorizationService.java:322)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:169)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$0(RBACEngine.java:151)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$4(CompositeRolesStore.java:193)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRole$5(CompositeRolesStore.java:211)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236)
	at org.elasticsearch.xcore@8.13.0-SNAPSHOT/org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$0(RoleReferenceIntersection.java:49)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:56)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.buildRoleFromRoleReference(CompositeRolesStore.java:243)
	at org.elasticsearch.xcore@8.13.0-SNAPSHOT/org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$1(RoleReferenceIntersection.java:53)
	at java.base/java.lang.Iterable.forEach(Iterable.java:75)
	at org.elasticsearch.xcore@8.13.0-SNAPSHOT/org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.buildRole(RoleReferenceIntersection.java:53)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRole(CompositeRolesStore.java:209)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:186)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:147)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:338)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:161)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:95)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticate(AuthenticatorChain.java:93)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:262)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:157)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:114)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:62)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:196)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:108)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.rest.action.RestCancellableNodeClient.doExecute(RestCancellableNodeClient.java:81)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:373)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.client.internal.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1129)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.client.internal.support.AbstractClient$IndicesAdmin.rolloverIndex(AbstractClient.java:1534)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.rest.action.admin.indices.RestRolloverIndexAction.lambda$prepareRequest$1(RestRolloverIndexAction.java:58)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:103)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.rest.SecurityRestFilter.doHandleRequest(SecurityRestFilter.java:94)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$0(SecurityRestFilter.java:85)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:169)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.lambda$authenticateAndAttachToContext$3(SecondaryAuthenticator.java:99)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticate(SecondaryAuthenticator.java:109)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticateAndAttachToContext(SecondaryAuthenticator.java:90)
	at org.elasticsearch.security@8.13.0-SNAPSHOT/org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:79)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:441)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:570)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:325)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:458)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:554)
	at org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:431)
	at org.elasticsearch.transport.netty4@8.13.0-SNAPSHOT/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.handlePipelinedRequest(Netty4HttpPipeliningHandler.java:128)
	at org.elasticsearch.transport.netty4@8.13.0-SNAPSHOT/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:118)
	at io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.codec@4.1.94.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
	at io.netty.codec@4.1.94.Final/io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
	at io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.codec@4.1.94.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
	at io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.codec@4.1.94.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
	at io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at org.elasticsearch.transport.netty4@8.13.0-SNAPSHOT/org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.forwardData(Netty4HttpHeaderValidator.java:194)
	at org.elasticsearch.transport.netty4@8.13.0-SNAPSHOT/org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.forwardFullRequest(Netty4HttpHeaderValidator.java:137)
	at org.elasticsearch.transport.netty4@8.13.0-SNAPSHOT/org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.lambda$requestStart$1(Netty4HttpHeaderValidator.java:120)
	at io.netty.common@4.1.94.Final/io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
	at io.netty.common@4.1.94.Final/io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106)
	at io.netty.common@4.1.94.Final/io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
	at io.netty.common@4.1.94.Final/io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
	at io.netty.common@4.1.94.Final/io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	at io.netty.transport@4.1.94.Final/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)
	at io.netty.common@4.1.94.Final/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.common@4.1.94.Final/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	... 1 more

gmarouli · 2023-12-22T09:59:02Z

@dakrone great catch! I missed a return statement :face_palm:. It only crashes when es is running with $ ./gradlew run, I will use this more from now on.

gmarouli · 2023-12-22T11:58:28Z

Missing something like this indicated that there were some tests missing. I added 2 more tests, one of them would have caught this.

gmarouli · 2024-01-02T08:33:31Z

@elasticmachine update branch

…pplies

gmarouli · 2024-01-05T07:47:25Z

@elasticmachine update branch

dakrone

This looks great, thanks Mary! I left two super minor comments. Thanks for breaking this up into separate pieces so it was easier to review also.

rest-api-spec/src/main/resources/rest-api-spec/api/indices.rollover.json

dakrone · 2024-01-05T15:40:53Z

server/src/main/java/org/elasticsearch/cluster/metadata/MetadataDataStreamsService.java

@@ -175,6 +219,36 @@ static ClusterState updateDataLifecycle(
        return ClusterState.builder(currentState).metadata(builder.build()).build();
    }

+    /**
+     * Creates an updated cluster state in which the requested data stream has the flag 'rollover_on_write' set to true.


The flag is passed in, so it's not necessarily always set to true, can you update this javadoc?

Good catch! Fixed

Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>

gmarouli added 4 commits December 12, 2023 09:29

Introduce lazy option in the rollover request

7b34c4e

Persist rollover needed flag in data streams

6faa9c5

Test lazy rollover applicability

00d112b

Add java rest test

bb558aa

gmarouli added >enhancement :Data Management/Data streams Data streams and their lifecycles labels Dec 12, 2023

elasticsearchmachine added v8.13.0 Team:Data Management Meta label for data/management team labels Dec 12, 2023

gmarouli and others added 3 commits December 12, 2023 10:37

Update docs/changelog/103309.yaml

92e0165

Reset rollover_needed when the ds rolls over

99af4f3

Add ER to the changelog

807837f

gmarouli marked this pull request as draft December 12, 2023 11:51

Small refactoring

5daf72f

gmarouli requested a review from dakrone December 12, 2023 13:26

gmarouli added 3 commits December 12, 2023 15:41

Add missing field from docs tests

a2488cb

Fix rollover cancellation test that was using concrete index

41f1cf3

Fix NPE when parsing bwc XContent

449f821

Merge branch 'main' into add-lazy-rollover-config

c49a6a5

gmarouli marked this pull request as ready for review December 13, 2023 11:01

gmarouli and others added 3 commits December 13, 2023 12:02

Update docs/changelog/103309.yaml

b145caa

Revert "Update docs/changelog/103309.yaml"

a09a074

This reverts commit b145caa.

Enhance the documentation

7cd426e

gmarouli commented Dec 13, 2023

View reviewed changes

Update docs/changelog/103309.yaml

ee2446a

dakrone reviewed Dec 13, 2023

View reviewed changes

Merge with main

b8afeb7

dakrone requested changes Dec 21, 2023

View reviewed changes

Add missing return statement

f0916e5

gmarouli requested a review from dakrone December 22, 2023 10:16

gmarouli added 3 commits December 22, 2023 13:54

Captured the failure in a test

b29e26a

Update rest-api-spec

9c6c6c1

Added extra test

3a0cd13

gmarouli added 2 commits December 29, 2023 11:30

Merge with main

4380e31

Merge branch 'main' into add-lazy-rollover-config

630fe2c

Merge branch 'main' into add-lazy-rollover-config

211e765

gmarouli mentioned this pull request Jan 2, 2024

Use RefCountingRunnable when creating indices in TransportBulkAction #103786

Merged

gmarouli added 4 commits January 3, 2024 11:30

Merge with main

244688f

Merge branch 'main' into add-lazy-rollover-config

614c45d

Rollover a data stream at the next write request when lazy rollover a…

7ef5da7

…pplies

Merge with main

afc73d3

Merge branch 'main' into add-lazy-rollover-config

f5cfc3b

dakrone approved these changes Jan 5, 2024

View reviewed changes

gmarouli and others added 3 commits January 8, 2024 10:12

Rephrase docs

f26ab13

Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>

Update javadoc

29ec8c5

Merge branch 'main' into add-lazy-rollover-config

3b80435

gmarouli merged commit 046cdea into elastic:main Jan 8, 2024
15 checks passed

This was referenced Jan 8, 2024

Update internal unconditional rollover calls to use lazy rollover #104083

Closed

Make use of lazy rollovers for invoked unconditional rollover elastic/kibana#174480

Closed

gmarouli mentioned this pull request Jan 11, 2024

Flag data stream as "needs a new index" to handle mappings updates where no documents are indexed #89346

Closed

gmarouli deleted the add-lazy-rollover-config branch August 20, 2024 07:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce lazy rollover for mapping updates in data streams #103309

Introduce lazy rollover for mapping updates in data streams #103309

gmarouli commented Dec 12, 2023 •

edited by dakrone

Loading

elasticsearchmachine commented Dec 12, 2023

elasticsearchmachine commented Dec 12, 2023

gmarouli commented Dec 13, 2023

elasticsearchmachine commented Dec 13, 2023

gmarouli Dec 13, 2023 •

edited

Loading

dakrone Dec 13, 2023

dakrone Dec 13, 2023

gmarouli Dec 14, 2023

elasticsearchmachine commented Dec 13, 2023

dakrone left a comment

dakrone Dec 13, 2023

dakrone Dec 13, 2023

dakrone left a comment

gmarouli commented Dec 22, 2023

gmarouli commented Dec 22, 2023

gmarouli commented Jan 2, 2024

gmarouli commented Jan 5, 2024

dakrone left a comment

dakrone Jan 5, 2024

gmarouli Jan 8, 2024

		DataStream.isFailureStoreEnabled() && args[11] != null && (boolean) args[11],
		DataStream.isFailureStoreEnabled() && args[12] != null ? (List<Index>) args[12] : List.of(),

Introduce lazy rollover for mapping updates in data streams #103309

Introduce lazy rollover for mapping updates in data streams #103309

Conversation

gmarouli commented Dec 12, 2023 • edited by dakrone Loading

elasticsearchmachine commented Dec 12, 2023

elasticsearchmachine commented Dec 12, 2023

gmarouli commented Dec 13, 2023

elasticsearchmachine commented Dec 13, 2023

gmarouli Dec 13, 2023 • edited Loading

Choose a reason for hiding this comment

dakrone Dec 13, 2023

Choose a reason for hiding this comment

dakrone Dec 13, 2023

Choose a reason for hiding this comment

gmarouli Dec 14, 2023

Choose a reason for hiding this comment

elasticsearchmachine commented Dec 13, 2023

dakrone left a comment

Choose a reason for hiding this comment

dakrone Dec 13, 2023

Choose a reason for hiding this comment

dakrone Dec 13, 2023

Choose a reason for hiding this comment

dakrone left a comment

Choose a reason for hiding this comment

gmarouli commented Dec 22, 2023

gmarouli commented Dec 22, 2023

gmarouli commented Jan 2, 2024

gmarouli commented Jan 5, 2024

dakrone left a comment

Choose a reason for hiding this comment

dakrone Jan 5, 2024

Choose a reason for hiding this comment

gmarouli Jan 8, 2024

Choose a reason for hiding this comment

gmarouli commented Dec 12, 2023 •

edited by dakrone

Loading

gmarouli Dec 13, 2023 •

edited

Loading