Calling a CCS index *:index_name requires remote_cluster_client when there are no remote clusters

[benwtrent]

Elasticsearch Version

8.14+

Installed Plugins

No response

Java Version

bundled

OS Version

any

Problem Description

When searching an index pattern that COULD refer to remote clusters, even though there are NO remote clusters configured, requires remote_cluster_client to be set.

Some solutions add "catch all" index patters in Kibana (e.g. *:metrics-*), this now requires all nodes to have remote_cluster_client role configured. For some customers, they prefer to minutely control the roles and now they cannot.

Steps to Reproduce

Create a cluster
Have no remote_cluster_client

Add some basic data and attempt to search

POST test/_doc
{
	"text": "quick brown fox, quick brown fox"
}


POST *:*/_search?error_trace
{
  "query": {
	"match_all": {}
  }
}

Logs (if relevant)

        "stack_trace": "java.lang.IllegalArgumentException: node [Benjamins-MacBook-Pro-2.local] does not have the remote cluster client role enabled
  at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.transport.RemoteClusterAware.groupClusterIndices(RemoteClusterAware.java:137)
  at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.transport.RemoteClusterService.groupIndices(RemoteClusterService.java:188)
  at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.ResolvedIndices.resolveWithIndicesRequest(ResolvedIndices.java:153)
  at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:344)
  at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:316)
  at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:121)
  at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:135)
  at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:54)
  at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:132)
  at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.MappedActionFilters$MappedFilterChain.proceed(MappedActionFilters.java:71)
  at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.MappedActionFilters.apply(MappedActionFilters.java:49)
  at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:132)
  at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.TransportAction.handleExecution(TransportAction.java:96)
  at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:59)
  at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:197)
  at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:106)
  at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.rest.action.RestCancellableNodeClient.doExecute(RestCancellableNodeClient.java:82)
  at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:140)
  at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.rest.action.search.RestSearchAction.lambda$prepareRequest$2(RestSearchAction.java:125)
  at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:144)
  at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.rest.RestController$1.onResponse(RestController.java:469)
  at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.rest.RestController$1.onResponse(RestController.java:463)
  at org.elasticsearch.security@9.0.0-SNAPSHOT/org.elasticsearch.xpack.security.rest.SecurityRestFilter.intercept(SecurityRestFilter.java:69)
  at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:463)
  at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:627)
  at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:306)
  at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:488)
  at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:584)
  at org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:461)
  at org.elasticsearch.transport.netty4@9.0.0-SNAPSHOT/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.handlePipelinedRequest(Netty4HttpPipeliningHandler.java:169)
  at org.elasticsearch.transport.netty4@9.0.0-SNAPSHOT/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:148)
  at io.netty.transport@4.1.115.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
  at io.netty.transport@4.1.115.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
  at io.netty.transport@4.1.115.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
  at io.netty.codec@4.1.115.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:107)
  at io.netty.codec@4.1.115.Final/io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:120)
  at io.netty.transport@4.1.115.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
  at io.netty.transport@4.1.115.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
  at io.netty.transport@4.1.115.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
  at io.netty.codec@4.1.115.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:107)
  at org.elasticsearch.transport.netty4@9.0.0-SNAPSHOT/org.elasticsearch.http.netty4.Netty4HttpAggregator.channelRead(Netty4HttpAggregator.java:52)
  at io.netty.transport@4.1.115.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
  at io.netty.transport@4.1.115.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
  at io.netty.transport@4.1.115.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
  at io.netty.codec@4.1.115.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:107)
  at io.netty.transport@4.1.115.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
  at io.netty.transport@4.1.115.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
  at io.netty.transport@4.1.115.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
  at io.netty.codec@4.1.115.Final/io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
  at io.netty.codec@4.1.115.Final/io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
  at io.netty.transport@4.1.115.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
  at io.netty.transport@4.1.115.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
  at io.netty.transport@4.1.115.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
  at io.netty.codec@4.1.115.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:107)
  at io.netty.transport@4.1.115.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
  at io.netty.transport@4.1.115.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
  at io.netty.transport@4.1.115.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
  at io.netty.transport@4.1.115.Final/io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
  at io.netty.transport@4.1.115.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
  at io.netty.transport@4.1.115.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
  at io.netty.transport@4.1.115.Final/io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
  at io.netty.transport@4.1.115.Final/io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
  at io.netty.transport@4.1.115.Final/io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
  at io.netty.transport@4.1.115.Final/io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689)
  at io.netty.transport@4.1.115.Final/io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652)
  at io.netty.transport@4.1.115.Final/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
  at io.netty.common@4.1.115.Final/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
  at io.netty.common@4.1.115.Final/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
  at java.base/java.lang.Thread.run(Thread.java:1575)
"

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)

[smalyshev]

The source of the issue seems to be in groupClusterIndices which does:

if (isRemoteClusterClientEnabled == false) {
     assert remoteClusterNames.isEmpty() : remoteClusterNames;
     throw new IllegalArgumentException("node [" + nodeName + "] does not have the remote cluster client role enabled");
}

However if we just remove the check, we may end up being hit by #115262 if there are no matching local indices, so it looks like these two needs to be fixed in concert or at least we should be aware of it when fixing that.