[Ingest Manager] Show the progress of updates

[mostlyjason]

Summary of the problem

We need a way to communicate to users the progress of updates to agents, whether they are agent binary upgrades or agent configuration changes. This is particularly important when there are thousands of agents since they may be in different states as a result of the upgrade. This will help administrators determine when all requested updates are complete and what agents need intervention to update / fix. We should show the status of updates when the user saves a change to an integration or agent configuration.

The status indicator should show the status of the agents where the update was requested. In other words, the user should the status of a filtered view of agents matching the agents that were selected, such as the agent policy. The user should be able to see how many agents have an update pending and how many are complete. It should be shown in the workflow of making the update, ideally right after the change is applied. This will allow the user to view progress as the update rolls out. The indicator can be closed or the user can change pages and the update should continue in the background.

It does not need to be a persistent way to view a history of changes, because that may be covered by a future audit logging and/or deployment tracking capabilities. If the user closes the page and returns later, the user will need to remember which agents they requested a change on to see if they are updated.

The Agent statuses should ideally match to the ones shown on the Fleet page to reduce cognitive load for users. #75236

Types of updates to track:

• Agent binary upgrades [Ingest Management] Add ability to upgrade agent binary in UI #68310
• Agent configuration updates
• Bulk actions [Ingest Manager] Add bulk actions #72712
• Package upgrades

User stories

• As a Fleet administrator, I need a way to track the progress of updates. I should be able to see the count of Agents in each state. This UI should update as the updates proceeds so that I can track progress and know when it's complete or reached a steady state.
• As a Fleet administrator, I need a way to determine which Agents were not updated due to an error or being offline so that I can take action to update or fix them. If an Agent has an error, I need a way to view Agent details and find the logs with the error message.

Stretch goal:

• As a Fleet administrator, I need a way to determine which hosts are sending data so that I can fix ones where the data is missing, and when I can proceed with visualization and analysis.
• As a Fleet administrator, I need a way to preview the data coming from my hosts so that I can identify and fix problems in my integration configuration or index template.

List known (technical) restrictions and requirements
None

Other
See wireframe on Whimiscal

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[mostlyjason]

@hbharding I updated the second and third paragraph with more clear requirements. I didn't mention the flyout as a requirement because its not clear we are taking that path, but its worth mentioning that it's a possibility.

[mostlyjason]

@hbharding I updated the description based on our discussions about fleet status yesterday

[hbharding]

Update

Recap of limitations

Based on previous discussions, we concluded that it will not be possible to track the status of individual deployments at this time.
To do so, we'll need to build some kind of "deployment object" that has a transaction ID which can be used to tie specific agents to timestamped actions. That way, a user could look at a bulk action and see a breakdown of how many agents succeeded, failed, not yet started, in progress, etc... This kind of functionality could potentially be useful for producing audit logs that describe who performed an action and when.

Without this capability, there are technical limitations that prevent us from showing accurate information about bulk actions. Additionally, while we may be able to hack a solution that could work for some bulk actions, this information would not persist between sessions and page reloads. If we were to show this information in a flyout after a user performs an action, it would not be possible to reaccess this information when the flyout closes.

Proposed interim solution

In lieu of these limitations, we can still improve the current experience by:

1. increasing the visibility of agent statuses when actions are performed
2. allowing administrators to scope (i.e., filter) status information to specific search criterea such as Agent policy
3. show a toast with a button that directs user to the Agents table when a change is made to a policy.

Before:

After:

A couple of things have changed with this proposal (ignore any naming discrepancies between the images):

• Changed the "status" column to use EuiBadge instead of EuiHealth
• Replaced status information in the header area with a colorful "status bar" visualization which includes a legend and breakdown of statuses by agent count. This visualization will update when filters are applied.

1. Use EuiBadge to show Agent status

Previously, it was challenging to recognize the agent status without reading the fine print. The 8px dot of EuiHealth made it difficult to distinguish the colors without looking closely. With EuiBadge, the colors are much more prominent and allow the user to quickly scan the table for status information. When an action is performed, the badge will change color (blue) and say "updating" until the action is complete. This emphasis on color should make a change in status from "updating" to "healthy" more apparent, whereas currently this change in status is very subtle.

Some notes:

• There are other places in Fleet such as the Agent Detail page where we've used EuiHealth to indicate agent status. We should update these instances to use EuiBadge so that the UI is consistent.
• We've discussed renaming the statuses so that "active" is renamed "healthy", "error" renamed "unhealthy", etc. These name changes may be out of scope for our initial implementation, however this shouldn't impact the design. We can keep our existing names, but still use them inside badges. [Fleet] Agent status improvements #75236

2. Status bar visualization

The status bar provides a breakdown of results for agent statuses across all pages of the table. Since we only show a limited number of agents at a time (10 by default), it is difficult for an administrator to have a complete picture of their agents' overall health. The current UI does not update the status count as filters are applied, whereas the proposed changes will add this kind of behavior.

When an administrator performs a bulk action like "upgrade agents", it is likely that they will want to limit the scope of the action to agents that share a particular policy. If the admin filters the list of agents for "Policy A" and then does a bulk select -> bulk action on all matching agents, they should notice the status bar change to a blue color to reflect that the agents are updating. As the agents update, the blue section of the status bar will gradually shrink to zero and be replaced by a status such as "healthy" or "unhealthy". This behavior will allow the user to have a better understanding of when a bulk action is in progress and eventually complete. Unlike a typical progress bar which transitions left to right from 0% to 100%, the status bar will indicate progress by counting down until the "updating" status reaches 0.

Some notes:

• The ordering of colors in the status bar is important. Agents that are "unhealthy" (yellow) or "updating" (blue) will appear before "healthy" (green) since this information is more interesting. It is assumed that most agents will be "healthy" (green). The proposed order is blue -> yellow -> green -> grey.
• If a filter is applied that returns 0 results, the status bar should be the same light grey color as "offline", and the counts in the legend should all indicate 0.
• When a user performs a bulk action from the agents page, a toast will appear with a spinner indicating that an action is in progress. This toast will disappear after 10 seconds.

3. Include button in toast messages that link to Agents table

It's easy to overlook, but when a user updates a policy that has agents enrolled (for example, by adding, editing, or removing an integration), they are in effect performing a bulk action. Currently, users are returned to the policy page when this type of action is performed; however there is no indication of agent status on this page. While this kind of information is something we might want to add to the policy page, for the time being, we can simply include a button in the toast message that will direct users to the agent table with a filter applied that matches the policy they were looking at.