[Security Solution] Rule Updates table has broken and users can't upgrade prebuilt rules

[banderror]

🚨🚨🚨 This is a blocker for the 8.9.0 release 🚨🚨🚨
I'm not escalating for now, hoping that we can fix it quickly.

Summary

It looks like when the 8.9.1 package with prebuilt rules was released, it broke our rule upgrade workflow. The Rule Updates table doesn't work anymore and doesn't show rules that can be upgraded:

Rule type updates are not supported

The error says Target version can't be of different rule type. This error means that in a newly installed security_detection_engine package there is a rule version in which the rule's type has been changed to a different one (e.g. from Custom query to EQL). In our new API endpoints for the rule upgrade workflow we don't support rule type updates, so in this case these endpoints return the error.

kibana/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/logic/diff/calculation/calculate_rule_fields_diff.ts

Lines 40 to 41 in 66fd6eb

	const BASE_TYPE_ERROR = `Base version can't be of different rule type`;
	const TARGET_TYPE_ERROR = `Target version can't be of different rule type`;

kibana/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/logic/diff/calculation/calculate_rule_fields_diff.ts

Lines 56 to 66 in 66fd6eb

	switch (current_version.type) {
	case 'query': {
	if (hasBaseVersion) {
	invariant(base_version.type === 'query', BASE_TYPE_ERROR);
	}
	invariant(target_version.type === 'query', TARGET_TYPE_ERROR);
	return {
	...commonFieldsDiff,
	...calculateCustomQueryFieldsDiff({ base_version, current_version, target_version }),
	};
	}

Proposal for fixing

The most important thing to do is to remove the 8.9.1 version of the security_detection_engine package from EPR as soon as possible. Since 8.9.0 is not yet released, no users in production should have it installed, so it should be safe to remove it. Also, changes introduced by elastic/integrations#6760 (and maybe other related changes, e.g. to the rule assets) will need to be reverted.

Then, we will need to revert rule type changes in the source detection-rules repo.

Then, a new 8.9.2 prerelease package needs to be built, pushed to EPR, and tested against Kibana by opening a PR in the kibana repo according to the release process. @jpdjere and @banderror will need to test this package in Kibana manually as well.

When the CI passes against the prerelease package, a release 8.9.2 package will need to be published in EPR.

After that, we will need to manually double-check that the 8.9.2 release package doesn't break the upgrade process in Kibana.

Then, new patch package versions will need to be released for older stacks. These patch packages must roll back rule type changes introduced in the latest patch versions:

I believe the patch package versions to build are: 8.8.7, 8.7.9, 8.6.9, 8.5.9.

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[terrancedejesus]

@banderror

Thanks for bringing this to our attention.

We removed the rule type change restriction after having a discussion with your team. We can turn this restriction back on if necessary after we fix the packages. Reference: elastic/detection-rules#2769

Do we know which specific rules caused this to error? I'll take a moment to find the offending rules shortly but wanted to ask first.

The 8.9.1 package was pushed out during our patch release for a PowerShell rule tuning last week. The other packages for this release were 8.5.8, 8.6.8, 8.7.8 and 8.8.6. These should all have the same offending rule if it were changed within the last 10 days or so.

[banderror]

@terrancedejesus Thank you for your quick response!

We removed the rule type change restriction after having a discussion with your team. We can turn this restriction back on if necessary after we fix the packages. Reference: elastic/detection-rules#2769

I guess this comment was correct on May 1st but this changed in 2 months during which we finalized the feature and made it ready for release in 8.9.0. We should have communicated this to you, I'm sorry that it wasn't done.

Yes, please turn the restriction back -- we won't support rule type updates in the new API endpoints and UI for prebuilt rules.

Do we know which specific rules caused this to error? I'll take a moment to find the offending rules shortly but wanted to ask first.

@jpdjere can debug this locally and find all the rules that are causing the bug.

The other packages for this release were 8.5.8, 8.6.8, 8.7.8 and 8.8.6. These should all have the same offending rule if it were changed within the last 10 days or so.

Okay, yeah, I think we should fix it for those packages as well by releasing new patch versions for them. Though it's not that critical compared to fixing the 8.9 package since in 8.8.x Kibana and below we do support rule type updates.

[terrancedejesus]

@banderror thanks for the follow up! No problem about the new restrictions, glad we can fix them before release. These are the only rule type changes I have found since the 8.3.1 release which was in June of 2022.

Query Type rules:

Rule ID	Name	Version	Query Language	Type	Change
2d8043ed-5bda-4caf-801c-c1feb7410504	Enumeration of Kernel Modules	104	eql	eql	PR Diff
a00681e3-9ed6-447c-ab2c-be648821c622	First Time Seen AWS Secret Value Accessed in Secrets Manager	206	kuery	new_terms	PR Diff
cd66a5af-e34b-4bb0-8931-57d0a043f2ef	Kernel Module Removal	104	eql	eql	PR DIff
d76b02ef-fc95-4001-9297-01cb7412232f	Interactive Terminal Spawned via Python	105	eql	eql	PR Diff

@jpdjere - From what @banderror said, the 8.9.1 package is the package having the issue since rule type conversions were fine up until 8.9 dev. Thus our next steps in TRADE would be.

• PR to tune these rules back to the previous logic and rule type.
• Version lock our rules and release an 8.9.2 package.

Questions:

• Do we need to remove the 8.9.1 package if it is ^8.9.0 and Fleet downloads the latest? This will be a patch release after all.
• Rather than change rules, are we able to keep the restriction off just as we have in 8.1-8.8? The ability to change rule types is very valuable to rule authors and we have several rule tuning PRs in the works to change rule types.

Example of rule type change from tuning from kuery (query) to kuery (new_terms):

I deployed an 8.9.0 stack from Staging and loaded the 8.9.1 package by default from Fleet and did not notice any errors that were originally reported. I also went and enabled these rules. Am I perhaps missing something specific that causes my rule loading to be not reporting errors or in a stable state? I assume the only difference is if it is an update rather than a new deployment?

Update

Installed an agent on Linux pointing to this stack and then simulated some activity to trigger an alert for one of these rules with rule type changes. Alert generated properly.

[jpdjere]

@terrancedejesus The issue shows up only when the user has any of the rules that had their type updated installed from a previous version of the package; and the 8.9.1 package is installed later.

(For example, user installs all rules from 8.5.8 and then the 8.9.1 package is installed in the background when the package becomes availalbe and the suer navigates to the Rules Management page)

Our new internal/detection_engine/prebuilt_rules/upgrade/_review endpoint runs an algorithm to return which rules have updates; but this same algo doesn't support updates of rule type, and thus returns 500.

I have reproduced it successfully by following the steps mentioned above. You can force the installation of a specific security_detection_engine package in Kibana with a config flag in kibana.dev.yml:

xpack.securitySolution.prebuiltRulesPackageVersion: '8.5.8'

[jpdjere]

@terrancedejesus Also, just wanted to confirm that I debugged locally using the procedure described above and our endpoint returned 500 when trying to calculate the diff for the 4 "offending" rules listed above (and no others).

[terrancedejesus]

@jpdjere - So the bug exists only on calculating the diff with rules that have changed rule types? And to re-create this we have to install an older package with the previous rule type, then upgrade to receive a package with a different rule type in 8.9+

Are there any drawbacks to adjusting the feature that calculates the diff to just accept rule type changes? Rather than remove packages, re-lock and release rules and re-introduce some noisy rules?

[terrancedejesus]

Just for additional information, when rule tunings are done those changes backport to the least compatible/supported stack in Detection Rules. So if a rule type was changed, that change backports from, for example, main->8.3. When we do our bi-weekly releases we release current-3 meaning we checkout each Detection Rules branch for 8.5, 8.6, 8.7, 8.8, build a package and push that out to EPR. The rule type change will then be present in every single OOB package to these stack versions. Not to mention, it will exist in older OOB packages for those stack versions as well.

Since the bug exists in the update workflow involving a new feature in 8.9, removing the 8.9.1 package or even attempting to release the previous "latest" packages for each stack does not remove the rule type change potentially as it could still be present in the package before those.

These packages are already in customer environments with these rule type changes in use.

To better understand where all these rule type diffs exist, I will spend some time reviewing each package from 8.4->8.8 to determine which packages have these rules and which rule types are present for each. This should help us identify the potential exposure of this bug.

From the AET side, would it be possible to explore any options available there that would maybe provide a stop-gap until 8.10 or further?

[terrancedejesus]

@jpdjere - Here is the breakout of rule type changes for these offenders across the last 31 ga package releases back to 8.5. <= 8.4 stack versions had the original rule type, therefore I did not include these as they are not affected. All of these rules can be found in OOB packages back to 8.0 stack versions. As you stated, if a user upgrades from 8.0 to 8.9 and the rule types differ, the bug would appear.

https://whimsical.com/rule-type-bug-7YjjjK2RS5SoS4oaRijc6L

Based on the information in the diagram. removing all packages is not ideal for this. On the flip side, if we remove the 8.9.1 package and push an update where 8.9.2 now has these rules with the original rule type, if someone upgrades from 8.5-8.8 to 8.9 the bug/error will still appear.

[xcrzx]

Are there any drawbacks to adjusting the feature that calculates the diff to just accept rule type changes? Rather than remove packages, re-lock and release rules and re-introduce some noisy rules?

We've had several discussions previously about the restrictions on changing rule types. As I understand it, these were introduced because we hadn't implemented that capability in the designs for the new workflow. That being said, it doesn't appear to be technically challenging to implement.

I'd suggest removing the rule type immutability restriction, rather than forcing a rule deprecation workflow to handle rule type changes. On top of that, removing invariants in the diff calculation algorithm (which is mostly unused in 8.9) seems much simpler than rolling back the rules package release.

[jpdjere]

Thanks @terrancedejesus for your work fleshing these changes out and helping us figure this out quickly.

Given the situation, we have decided to allow rule type changes in our update diff algorithm. This will increase complexity for us as we develop the algorithm to allow users to view diffs and customize their rules while updating them, but it's the only way we can prevent users from hitting this bug when updating to 8.9 (even if we release patches for all previous minor versions, we cannot force users to update all their "offending" rules before migrating).

This change will mean expanding our BE logic in the upgrade diffing algorithm to figuring out how to handle fields that have been removed or added during rule type changes, and in the FE, figuring out how to display these changes and how (or if) to allow the user to customize them.

The immediate work to do now to get rid of the issue is:

• remove the checks (invariants) in the upgrade diffing algorithm that assert that the rule type hasn't changed.
• add a check in the upgrade _perform endpoint that checks if the rule type has changed; if it has, allow the user to only choose to update to BASE, CURRENT, or TARGET version, but not MERGED. (The current milestone allows only upgrading to TARGET, but the check should be in place as of now)

cc @banderror @xcrzx

[banderror]

Thanks @jpdjere @terrancedejesus @xcrzx for finding the path forward.

[banderror]

We're working on a fix in this PR: #161247

[banderror]

@vgomez-el Could you please make sure the fix is verified by the team in the next BC?

[vgomez-el]

I have upgraded the prebuilt rules package from a 8.3.3 version to a 8.9.2 on 8.9 BC4 version successfully, and I haven't faced the bug. So I can validate the fix.