Support point in time cross cluster search

[dnhatn]

This commit integrates point in time into cross cluster search.

0a99df6 -> Closes #61790
Relates #61062

[elasticmachine]

Pinging @elastic/es-search (:Search/Search)

[jimczi]

LGTM

[jimczi]

Why is it needed ?

[dnhatn]

We need to disable ccs_minimize_roundtrips because we don't know how to merge a point in time from multiple sub-search requests.

[ywangd]

First of all, I have little knowlege about PIT's design and implementation. So my comments may be completely off. But in the spirit of learning, I am just going to ask it away :)

To perform CCS with PIT, it requires the user to have read privileges on a local index even when the target indices are all remote. i.e. if user has no index privileges on the local cluster, the call POST remote:point_in_time_index/_pit?keep_alive=5m results in the following error:

action [indices:data/read/open_point_in_time] is unauthorized for user [joe] on indices [], this action is granted by the privileges [read,all]

This behaviour is different from CCS with scroll or async_search, both of them do not require user to have any index privileges on the local cluster (as suggested by our docs), i.e. both following API calls work:

GET remote:point_in_time_index/_search?scroll=1h 

POST remote:point_in_time_index/_async_search?keep_on_completion=true&keep_alive=1h

(Note ccs with scroll has an issue where it requires local privileges after the initial open scroll call. But we consider it as a bug #61828).

Unless PIT's design is intentionally to be different on this front, consistency may be a better choice.

Another thing I noticed is that if only remote indices are used for creating PIT, the initial creation will success (provide user has local index privileges). But the subsequent usage results in NEP, i.e:

POST remote:point_in_time_index/_pit?keep_alive=5m
POST _search
{
  'query': {
    'range': {
      'created_at': {
        'gte': '2020-01-03'
      }
    }
  },
  'pit': {
    'id': "PIT_FROM_ABOVE_REQUEST",
    'keep_alive': '5m',
  }
}

// The above call results in error:

"Cannot invoke "org.elasticsearch.action.OriginalIndices.indices()" because "localIndices" is null"

Ping @albertzaharovits @tvernum for awareness

[albertzaharovits]

Thanks for testing and digging into things @ywangd ! These two are bugs for sure.

On the first point, about open-point-in-time requests that contain only remote indices, I think the fix should be simple add OpenPointInTimeRequest to the list in IndicesAndAliasesResolver#allowsRemoteIndices .

The other one should also be easy to fix, I would start from

elasticsearch/server/src/main/java/org/elasticsearch/action/search/TransportSearchAction.java

Line 294 in cbbe2d7

OriginalIndices localIndices = remoteClusterIndices.remove(RemoteClusterAware.LOCAL_CLUSTER_GROUP_KEY);

. Regular searches can have an empty list for "local indices" .

Can you take it home and raise the fix PRs as well ?

[ywangd]

Can you take it home and raise the fix PRs as well ?

I am happy to raise a following PR once this one is merged if that's what you are asking.

[dnhatn]

@ywangd Thank you for reviewing. I've pushed the fix for NPE in f83433b.

[ywangd]

@ywangd Thank you for reviewing. I've pushed the fix for NPE in f83433b.

@dnhatn Thanks for fixing the NPE. I'll raise follow-up one to fix the permission issue when target indices are all remote.