[Fleet] Packages with a large number of saved objects in them cause Kibana to crash

[xcrzx]

We have recently encountered an issue where Kibana crashes when installing a Fleet package that contains a large number of saved objects. The crash occurs during the installation process and seems to be caused by the deletion of the previous package version.

Steps to reproduce:

1. Install a Fleet package that contains a large number of saved objects (e.g. over 10,000) using POST /api/fleet/epm/packages/<package>/<version>.
   You could follow the steps from this ticket to generate a package with a large number of saved objects and install it.
2. Observe that Kibana crashes during the installation process.

Expected result:
The Fleet package should be installed successfully without crashing Kibana.

Actual result:
Kibana crashes during the installation process. Elasticsearch logs show dozens of warnings similar to this:

block until refresh ran out of slots and forced a refresh: [BulkShardRequest [[.kibana_8.7.0_001][0]] containing [delete {[.kibana_8.7.0][security-rule:d8fc1cca-93ed-43c1-bbb6-c0dd3eff2958:102.0.6]}] blocking until refresh]

During that time, all requests to Kibana fail with

{"statusCode":503,"error":"Service Unavailable","message":"connect EADDRNOTAVAIL 127.0.0.1:9200 - Local (0.0.0.0:0)"}

Notes:

This issue does not occur with smaller packages containing fewer saved objects.

The issue can be temporarily resolved by manually deleting the saved objects from the previous package version before installing the new one, but this is not a permanent solution.

APM logs show hundreds of DELETE requests sent in parallel, they seem to overflow Elasticsearch, making it unresponsive:

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[xcrzx]

Hey @joshdover, our team is experimenting with Fleet packages that could contain a significant number of saved objects of the security-rule type (potentially tens of thousands). Here are two PoC for reference:

1. [Security Solution] Historical rules packages PoC #145851
2. [Security Solution] PoC of the rule upgrade and installation workflows #144060

While working on those PoCs, we've encountered some limitations on the Fleet's side, presumably related to how package assets are tracked and then deleted. I.e., all installed package assets are listed in the installed_kibana property of the package's saved object:

kibana/x-pack/plugins/fleet/server/saved_objects/index.ts

Lines 261 to 267 in ab8dd04

	installed_kibana: {
	type: 'nested',
	properties: {
	id: { type: 'keyword' },
	type: { type: 'keyword' },
	},
	},

.

installed_kibana is used during a package upgrade to delete assets from the previous version. From what I see, listed saved objects are deleted one by one leading to performance problems mentioned in the ticket description. Would it be an option for Fleet to switch to the recently introduced bulkDelete saved object method to improve package upgrade performance?

cc @banderror

[joshdover]

Would it be an option for Fleet to switch to the recently introduced bulkDelete saved object method to improve package upgrade performance?

Yes we definitely should switch over now that this is available. We'd be open to accept a PR for that if your team has the time.

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)