[Fleet] Allow inspecting the raw agent status document to quickly diagnose unhealthy agents

[cmacknz]

Today the agent reports the detailed status of each component and input to Fleet. We are working to present this in an easy to understand way, but until that work is completed the only way to know why the agent is unhealthy is to collect and analyze the agent logs or run the elastic-agent status --output=json command on the agent host.

For example when an agent is unhealthy the user is presented with a "1 or more components in a failed state" message with no way to know what is actually wrong from the UI alone.

As a stop gap we should add a link to view the raw agent status reported to the Fleet UI, which will include the detailed state of each component and unit. This will allow advanced users to quickly view what is wrong as a stop gap solution. This is roughly equivalent to the output from the elastic-agent status --output=json command.

Specifically we should allow directly and easily viewing the agent status fields from the .fleet-agents index on the Agent Details page. Here is an example for a healthy agent implementing a log and system/metrics input.

          "last_checkin_message": "Running",
          "last_checkin_status": "online",
          "components": [
            {
              "id": "log-default",
              "units": [
                {
                  "id": "log-default-logfile-system-28261bdf-df61-4404-9db9-9f40cdd8f765",
                  "type": "input",
                  "message": "Starting: spawned pid '12814'",
                  "status": "STARTING"
                },
                {
                  "id": "log-default",
                  "type": "output",
                  "message": "Starting: spawned pid '12814'",
                  "status": "STARTING"
                }
              ],
              "type": "log",
              "message": "Healthy: communicating with pid '12814'",
              "status": "HEALTHY"
            },
            {
              "id": "system/metrics-default",
              "units": [
                {
                  "id": "system/metrics-default-system/metrics-system-28261bdf-df61-4404-9db9-9f40cdd8f765",
                  "type": "input",
                  "message": "Starting: spawned pid '12815'",
                  "status": "STARTING"
                },
                {
                  "id": "system/metrics-default",
                  "type": "output",
                  "message": "Starting: spawned pid '12815'",
                  "status": "STARTING"
                }
              ],
              "type": "system/metrics",
              "message": "Healthy: communicating with pid '12815'",
              "status": "HEALTHY"
            },
          ],

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[jen-huang]

This is already possible through the hidden debug UI at /app/fleet/_debug:

Granted, this UI is just a hard pull of the first 10k .fleet-agents and doesn't allow choosing an exact agent, but maybe it is enough for troubleshooting for now?

[cmacknz]

My intent is to make this much easier for users to access, so we wouldn't want it to be hidden (I had no idea /app/fleet/agents exited for example) and we would want to be able to easily see it for a specific agent.

The document structure here is almost an exact match for the output of the elastic-agent status --output=json command which users are already forced to rely on, we should just make it easy to see them same thing in the UI.

[amolnater-qasource]

Hi Team,

We have created 03 testcases for this feature under Fleet test suite at links:

• Validate View agent Json option is available under the Actions button.
• Validate that the Json file keeps getting updated as per update in Agent’s status.
• Validate that the user is able to download the Json file while the agent is under any status.

Please let us know if we are missing any scenario to be covered here.

Thanks!

[amolnater-qasource]

Hi Team,

We have executed test run for this feature on latest 8.8.0 BC3 kibana cloud environment at link:

• Inspecting the raw agent status

Status:
PASS: 03

Build details:
VERSION: 8.8.0
BUILD: 62994
COMMIT: 85b22d3

Hence we are marking this feature as QA:Validated.
Thanks