Cross Cluster Search limitations

[stacey-gammon]

There are some known limitations when using Kibana and Cross Cluster search to access data on an Elasticsearch cluster that is older than the Kibana version.

Limitations

Area	CCS Limitation	Expected versions broken
Discover	Data from incompatible cluster is silently lost	Kibana is >= 7.12 and remote cluster < 7.10
Maps	geotile_grid aggregation fails	Kibana is >= 7.0 and remote cluster is <= 6.8
Canvas	Elasticsearch queries will timeout	Kibana is >= 7.12 and remote cluster < 7.10
Canvas	SQL queries will timeout	All versions
Canvas	EQL queries will timeout	All versions
Logs	Complete data loss from both compatible and incompatible clusters	Kibana is < 7.12.1 & >= 7.11 and remote cluster < 7.10
Logs	Data loss from incompatible clusters	Kibana is >= 7.12.1 and remote cluster < 7.10
ML	Limitations, but user sees helpful error messages (full context)	Kibana is >= 7.12 and remote cluster < 7.10
SIEM	Timeline and Hosts page are missing data. More details	Most likely due to search fields option, so Kibana is >= 7.12.1 and remote cluster < 7.10
SIEM	Alerts do not generate data. More details	Most likely due to search fields option, so Kibana is >= 7.12.1 and remote cluster < 7.10

Details

Search fields options

Using the fields option in search requests in versions >= 7.11 will result in data not being returned from clusters <= 7.10. Areas broken in Kibana include Discover, Canvas (SQL,EQL), Logs and ML, SIEM. There is a workaround currently for Discover, bu turning the discover:searchFieldsFromSource toggle on (not the default).

Geo tile grid aggregations

These types of aggregations in the maps app and ML will fail if any clusters are <= 6.8 and Kibana is 7.0 and greater.

Progress & Status

7.12 Kibana -> 6.8 ES

Owner	Area	Status
@MadameSheema	SIEM	Not started
@qhoxie	Enterprise Search	✅ Done
@jasonrhodes	Observability - #96318	✅ Done
@cjcenizal	ES UI	✅ Done
@wwang500	ML	✅ Done - #93624 (comment)
@LeeDr	Kibana testing	✅ Done - #93624

7.11 Kibana -> 6.8 ES

Owner	Area	Status
@MadameSheema	SIEM	In Progress
@qhoxie	Enterprise Search	✅ Done
@wwang500	ML	✅ Done - #94965 (comment)
@jasonrhodes	Observability - #96318	✅ Done
@cjcenizal	ES UI	✅ Done
@LeeDr	Kibana testing	✅ Done - #95213

7.12 Kibana -> 7.11 ES

Owner	Area	Status
@MadameSheema	SIEM	In Progress
@qhoxie	Enterprise Search	✅ Done
@wwang500	ML	✅ Done - #94965 (comment)
@jasonrhodes	Observability - #96318	✅ Done
@cjcenizal	ES UI	✅ Done
@LeeDr	Kibana testing - #95214	✅ Done

[sophiec20]

From the ML UI + CCS perspective, much of the standard functionality works over CCS. However if the remote cluster is a prior version which does not contain a certain feature (for example an agg), then we aim to fail early with a useful error message. Functionality that is supported on both versions, continues to work.

Adding context to the list above, for 7.12 -> 6.8:

• Data frame analytics does not support CCS - User sees relevant error message. Error: An error occurred fetching analysis fields data. cannot retrieve data because index [7_12:kibana_sample_data_logs] does not exist. User can continue to create DFA jobs using index patterns that are not CCS. Ideally we want to fail earlier in the wizard with a better error message. Usability improvement [ML] need a better error message on DFA job wizard if using CCS #95257
• Transform geotile_grid grouby is not supported. - User sees relevant error message. Error: Failed to validate configuration: Failed to test query: Attempting to serialize [GeoTileGridValuesSourceBuilder] to a node with unsupported version [6.8.14]. User may continue using transforms providing this agg is not used in the configuration.
• Runtime Fields in search is not supported for both Transform and Anomaly Detection. - User sees relevant error message. Versions before 7.11.0 don't support [runtime_mappings] and search was sent to [6.8.14] . User can continue to create Transforms or Anomaly Detection jobs which do not contain runtime fields.
• Categorization job will hit an error on UI. - User sees error in setup wizard which prevents them from creating a categorization job. Selected category field is invalid No examples for this field could be found. As a workaround, user can create the anomaly detection job using API. The UI should not prevent the job from being created. Fix reqd [ML] Categorization validation failed if source index is from a remote CCS 6.8 cluster  #94737
• Transform source index view is empty if remote indices are on an old version cluster. - User can continue to step through the wizard and create the transform however this should be handled more gracefully. Usability improvement [ML] Transform source index view is empty if remote indices are on a remote CCS 6.8 cluster #95259

[maggieghamry]

@stacey-gammon can you please elaborate on what this icon means? 🚧 7.12 Kibana -> 7.11 ES - does this mean it doesn't work?

If upgrading remote clusters before the cross-cluster search cluster, should the existing CCS Kibana continue to work with the remote clusters throughout the process? (especially with clusters that can take hours to upgrade)

[stacey-gammon]

@maggieghamry, I've adjusted the main issue. I meant to indicate that testing is still in progress for that configuration. Please let me know if it is still confusing.

If you upgrade Kibana last, then everything should work. This is specifically testing situations where the Kibana version is ahead of the remote cluster version.

[mattkime]

We don't have any automated testing for cross cluster support, it would be nice if we did. I've dealt with cross cluster issues with index pattern creation.

[maggieghamry]

Thank you!

[cjcenizal]

ES UI findings

Testing steps:

1. Start local cluster ./bin/elasticsearch
2. Start remote cluster ./bin/elasticsearch -E transport.port=9400
3. On local cluster, connect to Remote Cluster with seed 127.0.0.1:9400
4. Create index on remote cluster curl -X PUT http://elastic:changeme@localhost:9201/my-index --data '{"settings":{"number_of_shards":1,"soft_deletes.enabled":true}}' --header "Content-Type: application/json"
5. Create follower index for my-index on local cluster
6. Index a document on the remote index curl -X POST http://elastic:changeme@localhost:9201/my-index/_doc --data '{"a": "b"}' --header "Content-Type: application/json"

Success: In all cases (7.12 Kibana -> 6.8 ES, 7.11 Kibana -> 6.8 ES, 7.12 Kibana -> 7.11 ES) I'm able to create a remote cluster and a follower index, and though the resulting index has yellow health it replicates documents.

[wwang500]

ML feature testing is done for both paths: 7.12 Kibana -> 7.11 ES: and 7.11 Kibana -> 6.8 ES:

7.12 Kibana -> 7.11 ES:

• All functions work as expected. ✅
• The two defects which were found during 7.12 Kibana -> 6.8 ES: [ML] Categorization validation failed if source index is from a remote CCS 6.8 cluster  #94737 and [ML] Transform source index view is empty if remote indices are on a remote CCS 6.8 cluster #95259 were not reproducible in this path test.
• Usability improvement [ML] need a better error message on DFA job wizard if using CCS #95257 is still observed.

7.11 Kibana -> 6.8 ES

• All functions work as expected. ✅
• Both two defects which were found during 7.12 Kibana -> 6.8 ES: [ML] Categorization validation failed if source index is from a remote CCS 6.8 cluster  #94737 and [ML] Transform source index view is empty if remote indices are on a remote CCS 6.8 cluster #95259 were still reproducible in this path test, which means both issues are not regression.
• Usability improvement [ML] need a better error message on DFA job wizard if using CCS #95257 is still observed.

[jasonrhodes]

@stacey-gammon I'll have some things to report on this next week from the Logs perspective.

If discover: discover:searchFieldsFromSource toggle is off (the default), data from the remote cluster will not be returned.

Just for whatever it's worth, the data will all return but the data from the incompatible clusters will be sparse and missing many fields. This will be the case for anyone using the fields API.

[jasonrhodes]

Logs UI testing is now complete:

7.11.2 Kibana -> 6.8.15 ES

• Log stream does not show any data
• Log alerting continues to work
• Log ML integration partially works (anomaly detection works, examples only shown for compatible clusters)

7.12.0 Kibana -> 6.8.15 ES

Same as 7.11.2 Kibana -> 6.8.15 ES

7.12.0 Kibana -> 7.11.2 ES

Logs UI works as usual (stream, alerts, and ML) for all data

---

Note: We expect that 7.12.1 and 7.13 to both work better with cross-cluster 6.8 ES, due to our finding and merging #94972, which will mean that the Log stream will begin to display all logs stored in compatible clusters, but log data from incompatible clusters (e.g. stored in the 6.8 cluster in this example scenario) will still not appear in the Log Stream UI.

Thank you to @weltenwort for completing this testing.

cc: @mukeshelastic @sgrodzicki

[stacey-gammon]

Aside from Fleet and Endpoint areas, SIEM has finished testing. There are issues with data not being displayed on the Hosts page and in Timeline, most likely due to the search fields option. In addition, there are issues with alerts generating data. Since 7.12 -> 7.11 has no issues, I also suspect this is due to the search fields option.

Aside from that EQL rules won't work, but this is expected.

More details:

• Overview of testing by versions
• More details into what exactly goes wrong

[qhoxie]

I think we can call this good for Enterprise Search. The solution only accesses its own "system" indices and CCS is not supported for those, so it shouldn't factor in. Additionally, Enterprise Search is recent enough that there isn't support for ES 6.x.

[damianpfister]

@sophiec20 - regarding your original comment in #94965 (comment), has anything changed in what we support/don't support specifically around ML?

If customers wish to use CCS in ML, from the Kibana UI, would it be viable or are the missing features sufficient to make it effectively challenging to consider creating ML jobs/datastreams based on local+CCS data?

Essentially I am trying to get an idea of what works and what doesn't based on:

From the ML UI + CCS perspective, much of the standard functionality works over CCS