[discuss][Fleet] Improve workflow for installing non-agent packages

[rw-access]

Describe the feature:
Not all packages relate to the agent or even policy. The new security_detection_engine package is the motivating use case (described below).

There's an extra page to "install the integration" that I find confusing. We're currently always prompted to

1. Select an agent policy
2. Configure the integration with a name
3. Choose a namespace (advanced options)

But if Fleet is more use as the delivery mechanism and the assets don't ever get sent to agent, these questions are moot and confusing.

Describe a specific use case for the feature:
The new security_detection_engine package installs prebuilt rules for the Detection Engine (in Security). These assets live entirely in Kibana and never interact with agent or policy.

Two possible approaches that come to mind

• Within the package definition, declare a new type of package, that's not integration and doesn't integrate with fleet.
• Have a flag within the package or auto-detect based off the asset types whether the second page is relevant.

Related concerns:

• Does the verbiage "install" always make sense? In this case, "install" just means that we create saved objects of the security-rule type. But nothing actually is changed for the end user; they still have to pull up the "Manage detection rules page" to finish the configuration and pull the rules into their space

cc @ruflin @spong @mostlyjason

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[elasticmachine]

Pinging @elastic/fleet (Feature:EPM)

[elasticmachine]

Pinging @elastic/fleet (Feature:Fleet)

[ruflin]

Long term, my preference is on defining a different package type. The better short term solution from my perspective would be to detect for example that no input templates are specified and because of this, not show add integration but instead install.

I quite like the "install" part. I would go so far that the way security rules are installed today should potentially also work for Kibana dashboards.

[mostlyjason]

I added this to @bradenlpreston priority list so he can rank it among other security priorities

[bradenlpreston]

@shimonmodi , @MikePaquette

Do we think that fleet is the best place to load the updated rules package? I'm not caught up on the rational, but would we be better served to have this workflow in the rules manager and then integrated into RAC in the future?

[rw-access]

Do we think that fleet is the best place to load the updated rules package?

This is a loaded question so I'll try to unpack and articulate how it all works for rules:

• we have a package security_detection_engine for EPR which contains all of the rules
• we can update this package between releases, without requiring the user to make stack updates (like they do today)
• as of [Security][Fleet] Install the security_detection_engine package automatically #97191, this package will be automatically enabled so that new rules, when published out-of-cycle will be automatically made available to the detection engine "manage detection rules" page. they won't be autoinstalled, the user will just see "updates available" and continue with the existing workflow
• fleet doesn't interface with the detection engine directly at all. Or the underlying framework (RAC). it just creates saved objects for the rules which make them available to the user. Inside the detection engine, they still have to choose which rules they want in their space

Since RAC is heavily in flux, I think this totally fine, and there really aren't any more options available to use today anyway. Maybe with more RAC development, new doors will open but for now are options are limited.

This ticket is referring to the fact that some packages in Fleet do not contain any agent-specific assets. So adding this integration to a policy is nonsensical. It's ultimately a UX issue, but the underlying plumbing seems to be okay.

This isn't as big of an issue for the next release, because the package will be automatically enabled anyway, so the saved objects will be updated as soon as updates are available. It's like a "staging area". So users can be completely ignorant to the Fleet aspect of the new rules updating feature.

If you want to read up on any more details for that effort, let me know and I'll be glad to point you to some resources, recording, issues, or hop on a call.

[mostlyjason]

@rw-access do you have any rules that need to be upgraded with the stack #89827? This might be needed if you have a code update that requires the user to update the package at the same time. Otherwise, your code would need be backwards compatible.

[rw-access]

We do have some handling for that. I think it's separate in scope from this issue, but I'll give you some context for how this is handled today by dual-publishing the rules:

• The security solution compiles some rules in as JSON files
• Fleet packages also contain rules
• When we install rules, we grab the most recent version of a rule that we can find in either location
• In the fleet package, we won't deliver rules to a stack that it doesn't support

If we get to the point where we can distribute packages into Kibana, then this will change. But either way, I believe that's distinct from this issue, which is mostly about the UX workflow non-agent packages.

Also, I only recently realized you can go to the settings tab to install the saved objects, which is exactly what I needed, and the "install integration" button doesn't make sense.