Import / export of rules and connectors as Kibana saved object Export/Import

[mikecote]

This issue summarizes the steps required to implement the import and export of rules and connectors in order to support the impending removal of legacy multi-tenancy. Import and export will occur from the saved object management page, so for this issue, we will not be enforcing RBAC restrictions on the rules and connectors. This will not preclude us from moving this functionality to the rule management page in the future and adding RBAC restrictions.

The proposed workflow will be to export and import rules and connectors in a disabled state, with any api key or secrets information nullified. After import, it will be up to the user to re-enable rules and connectors, with UX to prompt them to enter any required secrets for connectors.

Implementation steps

1. Add concept of disabled connectors

• API for enabling/disabling connectors #94106
• UI for enabling/disabling connectors #94119
• Alerting changes to handle disabled connectors #94128

2. Import/export of rules from saved objects management page

• Enabling export of rules and connectors #94151
• Enabling import of rules and connectors #94152

2021/05/13 Update

There is currently a bug with the SO export process that prevents connectors from being exported correctly when they are exported as part of a rule reference:

• Export hooks are not called when exporting referenced saved objects #100043

2021/07/01 Update

Import/export of rules and connectors has been merged into 7.14, with the caveat that security and stack monitoring rules are excluded from the export. We have the following issues to track the progress of aligning security and stack monitoring rule export with the framework export:

• Security - [Alerting] Include security rules in import/export #103600
• Stack monitoring - [Alerting] Include stack monitoring rules in import/export #103599

[elasticmachine]

Pinging @elastic/kibana-stack-services (Team:Stack Services)

[mikecote]

What happens when secrets are missing?

There is an issue open to allow end-users to access encrypted properties #56013. This would allow a few options for actions:

1. Export encrypted attributes and actions will work right away after import (auto generate new API key)
2. Export without encrypted attributes and have some interactive import where users have to input credentials for each action
3. Export without encrypted attributes and import actions as disabled until they are enabled with credentials

[mikecote]

Some additional challenges mentioned here: #60053 (comment).

• When moving alert SOs to another cluster, the API keys will not exist within Elasticsearch
• When importing alerts, task manager tasks will have to be created so these alerts run after importing
• Is is possible that the user wants to provide different credentials in their connectors when moving to another cluster
• The RBAC work we've done doesn't give the user direct access to the SOs
• As mentioned, usage of encrypted attributes (we may revisit this down the line for allowing those to be exposed now that we have RBAC)

[mikecote]

I have some ideas how some of the concerns above could be handled.

Connectors

Here we could rely on something like exposing connector secrets which could be included in the export. This will be a breaking change as users wouldn't expect such a change in a minor.

An alternative option could be to add a feature to connectors that allows users to disable them. This could behave the same as when connectors are disabled by the license where they don't run. We could then omit secrets and disable the connector in the export data. We could then prompt the user after import that further steps are necessary to enable the connector.

Alerts

I think a lot of the workflows could be solved by exporting / importing the alerts as disabled. It won't require an API key or a task to be scheduled, it won't rely on the importing user's privileges to make the alert work. Here as well we could prompt the user after import that further steps are necessary to enable the alerts and their related connectors.

[pmuellr]

I'm not a fan of exporting the secrets. It's basically a back-door to exfiltrate them. Convince an admin to export the actions, the admin not realizing they contain secrets. Kinda thing.

I wonder how hard it would be to allow connectors with required secrets to be imported WITHOUT them. I think we'd need to change all the secret fields to schema.nullable() in order for that to work, but then we lose our schema checks for those being required when being used as expected. We're already partially there with the hasAuth stuff that got added to email and webhooks, you could imagine a similar field justImported (obviously a better name), where the validation code would then allow the values to be null when that was true, and not allow them to be null when it's false, and we'd reset justImported to false on every edit. We could then also have more precise error messages if you used an imported connector that didn't have the secrets added: "you forgot to add the secrets back". And also highlight these in the connectors list.

[mikecote]

The justImported method is not a bad idea.

How do you envision showing users that they need to fix a connector before it can be used? I was thinking the enable / disable connector feature could work for this where the fields would have to be entered before enabling the connector somehow and the connectors list could filter on these. Or maybe that logic is handled when enabling the alert?

[pmuellr]

One option would to use enabled on a connector in lieu of justImported. enabled: false could be used in the test for a connector's validation that the secrets which would normally be required, can be null. We'd want to have the enabled checkbox in the connector edit form, so that when they click enabled on, we won't let them save till it's validated, which at that point would mean the normally required secrets must not be null.

Using enabled may have an additional benefit of allowing a customer to temporarily "disable" an action from running, without having to actually delete it from the alert.

Also remember, we can disable action types via config (and I think some other ways?), so that "is an action enabled" gets a little more complicated / confusing to the customer. But at least it's still a "single concept", vs having a separate, kinda weird thing about "this action was imported so not usable yet".

Next question, can we hook the export to set this field to false on export? Or hook the import? Or probably both, for safety?

I also wonder how discoverable this will be. It's really the connector, so I think you wouldn't want to be able to add a new action to an alert with a disabled connector - we can add some badge to the connector indicating it's disabled. But what about editing alerts with existing actions whose connectors are disabled? Is the action editable? Maybe just a banner in the action edit form indicating the connector is disabled, with a link to go right to it's editor?

[mikecote]

One option would to use enabled on a connector in lieu of justImported. enabled: false could be used in the test for a connector's validation that the secrets which would normally be required, can be null. We'd want to have the enabled checkbox in the connector edit form, so that when they click enabled on, we won't let them save till it's validated, which at that point would mean the normally required secrets must not be null.

Using enabled may have an additional benefit of allowing a customer to temporarily "disable" an action from running, without having to actually delete it from the alert.

Also remember, we can disable action types via config (and I think some other ways?), so that "is an action enabled" gets a little more complicated / confusing to the customer. But at least it's still a "single concept", vs having a separate, kinda weird thing about "this action was imported so not usable yet".

100% in agreement regarding this. We have disabled via config and disabled via license, there's some UX in the alerts view for this already and it could be "free" for the new disabled feature (I don't recall exactly how the UX goes, some may prevent saving the alert with disabled by license connectors).

Next question, can we hook the export to set this field to false on export? Or hook the import? Or probably both, for safety?

If we are looking at building our own import / export API due to RBAC limitations, we'll have full control over this flow :)

I also wonder how discoverable this will be. It's really the connector, so I think you wouldn't want to be able to add a new action to an alert with a disabled connector - we can add some badge to the connector indicating it's disabled. But what about editing alerts with existing actions whose connectors are disabled? Is the action editable? Maybe just a banner in the action edit form indicating the connector is disabled, with a link to go right to it's editor?

Good questions, I think these will be good discussions. These seem answerable if this becomes the overall import / export approach. And I think it would work for a minor release (pre 8.0).

[mikecote]

Below is a summary of what I believe is needed to achieve import / export in a 7.x release based on the conversations above. The following is based on integrating and leveraging saved objects management UIs and APIs for import / export (#82027).

NOTE: Saved objects management displays all the saved objects in the system regardless of user feature privileges.

1. Ability to specify custom id for alerts and actions #50210

Since the encrypted saved objects (ESO) plugin doesn't allow specifying custom IDs (see here) and the saved objects management allows overwriting on import, we'll need to support such scenario.

We should verify that this step is needed and coordinate with the Kibana Security team to revive #42762. The work specific to the alerting APIs to support this could be split into a follow up issue or done at the same time.

2. Ability to enable / disable connectors

A new feature that could be useful to users who want to stop all executions of a connector. This is similar to using xpack.actions.enabledActionTypes in kibana.yml but dynamic and at the connector level.

The usage of this for import / export would be to export all the connectors as disabled and without secrets and have some UX when users try to enable connectors with missing secrets. Alerts can still reference disabled connectors but they will not execute. Exact UX to be determined.

3. Integrate alerts and connectors to saved objects management #82027

This would be an effort shared with the platform team to have alerts and connectors available in the saved objects management UIs. In the future, we can leverage copy to space and share across spaces feature (and/or integrate within our UIs).

For import / export to work, we will need custom logic to export alerts and connectors. Both would "disable" the alert / connector on export. The removal of secrets should already be done by the find API ESO wrapper.

[mikecote]

If Kibana B already had auto-created stack monitoring rules with the _id from Kibana B, there would be duplicates.

@ymao1 keep me honest 🙂 @jasonrhodes in this scenario, I think the user will be made aware the rules already exist and ask if they wish to overwrite them or create duplicates. There may be the option for the user to ask for new IDs on import, at that point we wouldn't be able to detect if there are duplicates afterwards.

Is it still possible to import with secrets via the API? I'd expect to be able to store the secrets for my connectors in something like Vault and pull them out for use in importing with some API calls.

@smith at this time no, we've been cautious about breaking the saying that "once secrets are stored in Kibana, we don't expose them to users afterwards". We've been thinking about breaking that saying here but have not found a valid reason to break the behaviour, especially not in a minor release.

For now, we'll need to restrict Stack Monitoring rules from import/export until we solve #91145 at least, and possibly #100133 as well. Then we'd be fine to just let them be imported/exported as all other rules.

@jasonrhodes is it right to assume the work in those issues will be completed in 7.14 / 7.15? I'm asking to ensure we have a way for users to move their customized stack monitoring rules from one environment to another by 7.16.

[ymao1]

If Kibana B already had auto-created stack monitoring rules with the _id from Kibana B, there would be duplicates.

@ymao1 keep me honest 🙂 @jasonrhodes in this scenario, I think the user will be made aware the rules already exist and ask if they wish to overwrite them or create duplicates. There may be the option for the user to ask for new IDs on import, at that point we wouldn't be able to detect if there are duplicates afterwards.

@mikecote Unfortunately, this is not the case. Since stack monitoring rules are autocreated without specifying a static _id, it is very likely (pretty much guaranteed) that the auto-created stack monitoring rules in Kibana A have different _ids than the auto-created stack monitoring rules in Kibana B, so importing from A -> B will not provide that "overwrite or create duplicates" popup.

What could happen is if stack monitoring rules are imported from A -> B before the stack monitoring rules are autocreated in Kibana B. Then the first time user visits stack monitoring in Kibana B, the auto-creation should be skipped because there are already (imported) rules of the stack monitoring types in B.

[mikecote]

If Kibana B already had auto-created stack monitoring rules with the _id from Kibana B, there would be duplicates.

@ymao1 keep me honest 🙂 @jasonrhodes in this scenario, I think the user will be made aware the rules already exist and ask if they wish to overwrite them or create duplicates. There may be the option for the user to ask for new IDs on import, at that point we wouldn't be able to detect if there are duplicates afterwards.

@mikecote Unfortunately, this is not the case. Since stack monitoring rules are autocreated without specifying a static _id, it is very likely (pretty much guaranteed) that the auto-created stack monitoring rules in Kibana A have different _ids than the auto-created stack monitoring rules in Kibana B, so importing from A -> B will not provide that "overwrite or create duplicates" popup.

What could happen is if stack monitoring rules are imported from A -> B before the stack monitoring rules are autocreated in Kibana B. Then the first time user visits stack monitoring in Kibana B, the auto-creation should be skipped because there are already (imported) rules of the stack monitoring types in B.

Ohh that makes total sense, I didn't think about that. I withdraw that comment 🙈

[jasonrhodes]

@jasonrhodes is it right to assume the work in those issues will be completed in 7.14 / 7.15? I'm asking to ensure we have a way for users to move their customized stack monitoring rules from one environment to another by 7.16.

@mikecote 7.14, no. 7.15, very hard to say. @ravikesarwani this is another thing driving the need to align SM alerts. This one is tricky because we probably have to solve most of the tickets in this meta ticket/epic before this will work without causing problems.

cc: @sgrodzicki / @paulb-elastic

[ravikesarwani]

FYI @mukeshelastic for awareness.
I agree @jasonrhodes. It's very important to align the SM alerts to work like other alerts so that its a common experience for solution users, doesn't break as framework evolves and able to utilize new things that gets added to the framework. As we have discussed I am on board with the changes that you have proposed in the Meta issue.

[mikecote]

@jasonrhodes if it's not 7.15, is it 7.16 or 8.x? We are trying to decide what to do with the import/export for edge cases and have two options that require time:

1. We let solutions that have edge cases build their own import/export experience to ensure user's customizations like rule actions can move to another Kibana. We need a heads up so we can work with @elastic/kibana-core team to exclude your rule saved objects (if possible, that's also TBD).
2. We have solutions fix their issues before 7.last (7.15 hopefully to give time for remaining steps) and have a unified import/export place for our users to go to when moving away from multi-tenancy.

@spong we have planned to work on supporting customization of references. Would that be the only change requested on your end to have an initial unified import/export experience in 7.last? or still TBD?

[spong]

@spong we have planned to work on supporting customization of references. Would that be the only change requested on your end to have an initial unified import/export experience in 7.last? or still TBD?

Yup! Supporting references (#87992) is the only dependency we're tracking at the moment. With that change we'll be able to update rules to reference any related exception lists and better manage the import/export flow (along with perf enhancements around the Exceptions Table and bulk export (#85173)).

That said, and as I see was discussed a bit above, we still need to determine how we're going to support the export/import of actions. If a user has 500 rules in their cluster, each with their own exception list/value list, and custom action, we have the means of restoring the rules, will work a solution for bulk import of exceptions, but don't currently have a method for backing up/restoring actions (or the ability to bulk assign actions, which could be a suitable substitution). Are actions still planned as part of this effort, or just focusing on rules/connectors for the time being?

[jasonrhodes]

@jasonrhodes if it's not 7.15, is it 7.16 or 8.x? We are trying to decide what to do with the import/export for edge cases and have two options that require time:

@mikecote totally understand, I'll leave my opinion here and let @mukeshelastic and @ravikesarwani weigh in as well. I think your option (2) is the right option, and that I don't see us having bandwidth to build a custom import/export system for SM rules anyway. That said, I don't think we need to push our timeline problems onto you, either. There may need to be a way to exclude certain rules from the import/export flow temporarily, and the result would be "Stack Monitoring rules are not importable/exportable at this time, please visit the UI in your new cluster to recreate your rules". None of them have that much customizability options, so I don't think this would be a massive problem.

I do think we have a chance of getting our alerts aligned by the time you make this live in 7.x, but I definitely can't guarantee it, and weighed against the other things we are trying to deliver, making sure we avoid the slightly unfortunate problem I describe above sounds like a very low priority, relatively, from the Stack Monitoring perspective.

[ymao1]

That said, and as I see was discussed a bit above, we still need to determine how we're going to support the export/import of actions. If a user has 500 rules in their cluster, each with their own exception list/value list, and custom action, we have the means of restoring the rules, will work a solution for bulk import of exceptions, but don't currently have a method for backing up/restoring actions (or the ability to bulk assign actions, which could be a suitable substitution). Are actions still planned as part of this effort, or just focusing on rules/connectors for the time being?

@spong rule actions are currently stored as SO references (referencing the connector that is backing the action) within the rule saved object, so when a rule is exported through the SO management UI with export referenced objects checked, both the rule, with its actions & the backing connector are included in the export. Is that what you are asking about, or is there another type of action?

[spong]

@spong rule actions are currently stored as SO references (referencing the connector that is backing the action) within the rule saved object, so when a rule is exported through the SO management UI with export referenced objects checked, both the rule, with its actions & the backing connector are included in the export. Is that what you are asking about, or is there another type of action?

Perfect, that's exactly what I was curious about. So users will be able to backup all their actions via connectors, so on the security side we'll just need to ensure the security rules maintain this reference as well so they can be re-associated on import. Thank you @ymao1! 🙂

[ymao1]

So users will be able to backup all their actions via connectors

@spong Just to make sure we're on the same page: connectors are stored as saved objects and actions are stored within a rule saved object, with a reference to the connector saved object for each action (along with params specific to that action). So when you say backup all their actions via connectors, are you saying users should use the SO management UI to export their connectors and security solution will be updating their rule export to include the configured actions?

Also, please note that connectors will be exported without their secrets fields because secrets are encrypted and we can't guarantee that the encryption key will be the same if export occurs in Kibana A and then imported to Kibana B. So there is some manual intervention required to re-enter connector secrets after import.

[spong]

So when you say backup all their actions via connectors, are you saying users should use the SO management UI to export their connectors and security solution will be updating their rule export to include the configured actions?

My apologies, meant rules not connectors. So if we end up disabling the export of Security Rules within SO Management we'll need explicit support for including actions as part of the security Solution Rule import/export. I'll capture this in the issue I'm writing (#100956) to ensure we support actions importing/exporting on the security side. If we have success with #99741 / #87992, we might just be able to leverage the SO Management UI for Security Rules/Exceptions/Actions (value lists will still need to be managed via the dedicated Security Solution UI though).

[ymao1]

@spong Gotcha. Thanks for the clarification!

[mikecote]

@spong

Yup! Supporting references (#87992) is the only dependency we're tracking at the moment. With that change we'll be able to update rules to reference any related exception lists and better manage the import/export flow (along with perf enhancements around the Exceptions Table and bulk export (#85173)).

Awesome, this is good to know!

That said, and as I see was discussed a bit above, we still need to determine how we're going to support the export/import of actions. If a user has 500 rules in their cluster, each with their own exception list/value list, and custom action, we have the means of restoring the rules, will work a solution for bulk import of exceptions, but don't currently have a method for backing up/restoring actions (or the ability to bulk assign actions, which could be a suitable substitution). Are actions still planned as part of this effort, or just focusing on rules/connectors for the time being?

With this part, I believe @ymao1 covered this but wanted to make sure. I believe you mentioned that your actions are stored as separate saved-objects? If so, you'd just need to add the reference to them in the rule (just like exceptions and value list).

@ymao1 is there a change needed on downstream/referenced saved-object types to include themselves in the export? or are they included automatically without that saved-object type marking itself exportable?

---

@jasonrhodes

None of them have that much customizability options, so I don't think this would be a massive problem.

I've seen a lot of user requests to be able to modify these SM rules and we've shown where to go for this. Users can modify things like rule actions? If so, they can customize anything they want there and wouldn't have an easy way to transfer that (they would have to go through each rule and re-write their customizations and templates for each rule action).

I do think we have a chance of getting our alerts aligned by the time you make this live in 7.x, but I definitely can't guarantee it, and weighed against the other things we are trying to deliver, making sure we avoid the slightly unfortunate problem I describe above sounds like a very low priority, relatively, from the Stack Monitoring perspective.

Based on this, I'll assume you are ok with the user experience risk that I stated above and we'll go ahead and talk to @elastic/kibana-core team to explore options of excluding SM rules from export (cc @ymao1).

[pgayvallet]

Based on this, I'll assume you are ok with the user experience risk that I stated above and we'll go ahead and talk to @elastic/kibana-core team to explore options of excluding SM rules from export (cc @ymao1).

Let's continue the discussion about this in #99680 then. There are a lot of implications that need to be discussed.

[ymao1]

@ymao1 is there a change needed on downstream/referenced saved-object types to include themselves in the export? or are they included automatically without that saved-object type marking itself exportable?

@mikecote @spong Any downstream saved object types will need to be marked as importable and exportable to be exported through the SO management UI. This has been done for connectors, so rules that contain a connector (action SO type) in its references array will automatically have the connector exported as well. When #87992 is resolved and rules have exceptions and value lists in their references array, those SO types will need to be marked as importableAndExportable in order to be exported with its referencing rule.

[jasonrhodes]

Based on this, I'll assume you are ok with the user experience risk that I stated above and we'll go ahead and talk to @elastic/kibana-core team to explore options of excluding SM rules from export (cc @ymao1).

Yeah, I don't think we have any other option, unfortunately. As I said, we hope we get this sorted before then, but if not, this will have to be the fallback. cc: @ravikesarwani / @mukeshelastic / @sgrodzicki / @paulb-elastic for visibility.

[gmmorris]

As we are carrying this Epic over from 7.14 to 7.15 I have updated the candidate labels.

This is in order to enable the work needed for the two remaining issues that enable Import/Export in Security Solution and Stack Management.