[EPM] Auto update packages

[neptunian]

If the user has not turned off auto updating, check for updates and try to update

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[neptunian]

@ruflin @hbharding is this happening in the background silently like when we install the default packages or are we wanting to notify the user that its happening? and when does it happen? I am thinking it should be part of the setup endpoint which is when they’ll land anywhere in the ingest manager and we check if the default/required packages are installed and install them

[ruflin]

If we do it in the background without a user logged in, we need some user credentials stored somewhere which I think we should not do for now.

Lets assume we would go with the setup approach. This would basically check every time someone clicks around in Ingest Manager? How often should it happen?

The part I'm more curious about is "how" does it figure out there are updates? But probably we should take this to a separate issue.

[neptunian]

When i said "background" I meant indicating to the user that packages are being updated vs doing it quietly. Whether or not we want the user to be aware of updating and for it to take place at a certain event (navigate to integrations tab) vs unknown to them, checking for updates every so often.

@ruflin I was thinking to check only when they visit the Integrations page or Ingest Manager (like the setup endpoint). At some point we can track the request and let them know that packages are being updated and when its finished a notification that pops up for each package that finishes updating. "Nginx has just been updated to 1.0.0!" Otherwise i'm not sure how they would know when that happened if it was happening silently behind the scenes. I'm thinking it's unlikely that there would be a large amount of packages updating at once with a lot of notifications popping up. Would be curious to hear @hbharding 's thoughts .

[ruflin]

What happens if a user closes the Kibana tab in the middle of an upgrade? Will this stop all processes?

[jonathan-buttner]

I had slack discussion with @neptunian about how auto updating of default packages (base, system, and endpoint) can be disabled using the opt out functionality.

I'm wondering if this will complicate releasing new features in the security solutions app. I can imagine the following scenario:

Let's assume that the auto update and opt out functionality is implemented in 7.9.

A user gets kibana 7.9 and they install an endpoint on one of their hosts using fleet. They then opt out of automatic updates for all packages including endpoint.

We release 7.10 which adds a new feature that relies on a new field in our endpoint package's mapping, and relies on the endpoint binary sending some new data.

A user wants the new feature so they upgrade their stack to 7.10, and upgrade the endpoint running on their host. The feature won't work though because they haven't updated their endpoint package.

This may get more complicated depending on if we turn dynamic mappings off for endpoint in that we might not be able to find documents that were sent by the endpoint for 7.10 even after the user installs the new endpoint package because the mapping wasn't applied yet.

I'd also need to think through scenarios where we have a breaking change that needs to be coupled with a stack upgrade and endpoint binary upgrade and how we'd handle that.

Would it cause problems to not allow a user to opt out of automatically upgrading the default packages? Or maybe just the endpoint package?

@ruflin @james-elastic @nnamdifrankie @seanelastic any thoughts?

[jonathan-buttner]

I followed up with @ruflin and we think the best solution is to have our app always make a request to install the latest endpoint package regardless of whether the user has opted out of auto upgrade of packages.

[paul-tavares]

@jonathan-buttner so we would do a single call during setup() or start()?
If so, that still would not handle package updates that may occur out-of-band with Kibana, right?

[paul-tavares]

One other thing to highlight, which I am not yet sure if its an issue or has any impact. We recently (today) added a "Create Policy" button in endpoint, which routes the to integrations/endpoint-0.3.0/add-datasource. The version number used to create this URL is obtained by doing a search against the EPM API - I think that API returns only the latest package information from the registry and not the latest that is installed. This might not be obvious to the user, but essentially we are always using the latest package when adding a new datasource, and I think if that package version is not installed when the user attempts to create the datasource, it will be done automatically.

Again maybe this is ok, but just wanted to highlight it.

What about Migration? is that covered in this Issue?
There maybe cases where existing Datasources need to migrated when a new package version installed, right?

[jonathan-buttner]

@jonathan-buttner so we would do a single call during setup() or start()?
If so, that still would not handle package updates that may occur out-of-band with Kibana, right?

Yeah I think we're going to have instruct the ingest manager to find and install the latest endpoint package in more than just the start() or setup() since that only happens when they log in. I wonder if it's feasible for use to release a package and for the user to still be logged in. Seems like a weird race condition or edge case.

The other option is to have our UI or backend instruct the ingest manager to do it whenever the user visits a page or hits our backend api.

One other thing to highlight, which I am not yet sure if its an issue or has any impact. We recently (today) added a "Create Policy" button in endpoint, which routes the to integrations/endpoint-0.3.0/add-datasource. The version number used to create this URL is obtained by doing a search against the EPM API - I think that API returns only the latest package information from the registry and not the latest that is installed. This might not be obvious to the user, but essentially we are always using the latest package when adding a new datasource, and I think if that package version is not installed when the user attempts to create the datasource, it will be done automatically.

Again maybe this is ok, but just wanted to highlight it.

Thanks for pointing that out. I think that's fine.

What about Migration? is that covered in this Issue?
There maybe cases where existing Datasources need to migrated when a new package version installed, right?

I think this issue talks about some of the datasource issues: https://github.com/elastic/integrations-dev/issues/25

One thought @ruflin had was doing the installation during a stack migration we're not sure if we'll have the right permissions to do that.

[ph]

After discussing with @ruflin, we believe we should not turn it on by default in beta releases.

[jonathan-buttner]

@neptunian and I had a slack conversation about how to move forward with the changes that came out of the brainstorming session. I'll post some of the notes here:

Cloud

I confirmed with the cloud team that it is possible to make API calls with a Kibana superuser after a stack upgrade. Still waiting to hear how we can move forward with this.

Question: Should this be done only for the endpoint package, endpoint package and system package, or all packages that have auto upgrade enabled?

On Prem

Things are more difficult for the on prem scenario. The short term solution will be: when a user opens the Security Solution or Ingest Manager apps, check to see if the latest endpoint package is installed.

If it is not, check to see if the user is a superuser and install the latest endpoint package.
If the user is not a superuser, display some sort of notification asking for a superuser to login and navigate to the Security Solution or Ingest Manager apps.

For other packages that have auto upgrade enabled we'll need to follow the same solution but it will only be checked in the Ingest Manager app.

Notification Flow

This flow would run regardless of whether Kibana was in the cloud or on-prem. If it is in the cloud, I don't expect we'd ever display a notification though because the upgrade would happen using the cloud hook.

Notes:

• If the user dismisses the notification, we'll probably want to notify them again sometime in the future

Endpoint binary upgrades

The endpoint package should be installed before a user can upgrade the endpoint binary. I'm not familiar with how the endpoint binary flow is going to work but ideally we'll make the user wait until the endpoint package has finished installing before the user can move forward with upgrading the endpoint binary through fleet.

[jonathan-buttner]

@oatkiller @kqualters-elastic @bkimmel @paul-tavares @michaelolo24 @XavierM @andrew-goldstein

What are your thoughts on implementing the notification flow mentioned above?

The brief background is that we have an endpoint package that we need to keep up to date in Kibana. Ideally we'd do this installation whenever the stack is upgraded (see the cloud scenario). The on-prem does not having a post stack upgrade hook though, so we need some other mechanism to get a privileged user to log in and navigate to our Security Solution app. The installation of the package must be done by a Kibana superuser. The package contains things like mappings, transforms, pipelines, visualizations, and other components that are installed in Elasticsearch and Kibana.

[bkimmel]

@jonathan-buttner Per our discussion in office hours, you may want to "capture intent" (with some kind of click explaining what's happening) before you "install the latest endpoint package" from the super-user account.

[ruflin]

@jonathan-buttner Lets focus for now only on endpoint and potentially system package, for the others we will follow up.

[paul-tavares]

@oatkiller @kqualters-elastic @bkimmel @paul-tavares @michaelolo24 @XavierM @andrew-goldstein

What are your thoughts on implementing the notification flow mentioned above?

Showing the Notification is ok with me, but we should ensure that the "notify again in the future" does not become a pain to the user. This would be done across all of the sub-plugins of Security Solution correct?

I'm also thinking that the "auto-install" of the package might be need further UX - especially if for some reason it fails.

All that being said - if a superuser does not login and visit any security solution page or Ingest prior to a regular user, then I think we should also be trying to ensure that the UI can work with two possible running Kibana states (one with the old package and another with the new package). I think for 7.10 and for endpoints list specifically (that will require a transform) that is likely going to be the case for the API (/endpoint/metadata, cc: @nnamdifrankie ).

cc: @kevinlog

[kevinlog]

@paul-tavares

then I think we should also be trying to ensure that the UI can work with two possible running Kibana states (one with the old package and another with the new package). I think for 7.10 and for endpoints list specifically (that will require a transform) that is likely going to be the case for the API (/endpoint/metadata, cc: @nnamdifrankie ).

Yes, this would have to be the case for backwards compatibility since we can't rely on the package being upgraded. We'd need to similar flag in the UI to hide the KQL bar since it would also require the new package. I'll create a new ticket for this.

[jonathan-buttner]

Showing the Notification is ok with me, but we should ensure that the "notify again in the future" does not become a pain to the user.

Yeah I agree.

This would be done across all of the sub-plugins of Security Solution correct?

That's correct.

I'm also thinking that the "auto-install" of the package might be need further UX - especially if for some reason it fails.

Yeah after talking with it over more I think we're leaning towards having the user initiate it even if they have the right permissions. So it wouldn't be automatic. I plan on bringing it up at the sync tomorrow and we can talk about it more.

Yeah I agree about the two different Kibana running states, we'll have to handle that.

[paul-tavares]

@jonathan-buttner sounds good.

If we decide to move forward with an explicit user action, I would propose centralize the page in the Ingest plugin (perhaps a browser route (ingest_manager/upgrade), since it could then be re-used by any future plugin that may need the same thing. This Ingest "upgrade" page can be coded to allow us (Security Solution) to "tell it" to navigate back to security once the upgrade is done (we do this today with a few of the endpoint management flows). This could also simplify the implementation in security solution around user interactions.