[Fleet] Upgrade details telemetry #173356

juliaElastic · 2023-12-14T10:37:20Z

Summary

Added upgrade details telemetry, publishing to fleet-agents index in telemetry cluster, each bucket as separate documents.
Implemented by doing a multi_terms aggregation to group the same target_version, state, error_msg values together.
Do we also want to include the agent count in each bucket in the telemetry event? @jlind23 @ycombinator

Note: since this task runs every hour, it will most likely capture the UPG_FAILED states, since the other (success) states are temporarily on the agent docs, and removed if the upgrade is successful.

E.g. 2 docs like the below become one telemetry event

// .fleet-agents
   upgrade_details: {
            target_version: '8.12.0',
            state: 'UPG_FAILED',
            metadata: {
              error_msg: 'Download failed',
            },
          },

// telemetry event
{
      target_version: '8.12.0',
      state: 'UPG_FAILED',
      error_msg: 'Download failed',
    }

To verify:

start kibana 8.13-SNAPSHOT locally
set an invalid agent download source in Fleet Settings
enroll an agent version 8.12-SNAPSHOT
upgrade to 8.13-SNAPSHOT with the API

POST kbn:/api/fleet/agents/<agent_id>/upgrade
  {
    "version": "8.13.0-SNAPSHOT",
    "force": true
  }

wait 15m so that the upgrade goes to failed state
wait up to 1h for the telemetry task to run (speed up locally by setting a shorter interval in FleetUsageSender in kibana)
verify in debug logs:

[2023-12-14T14:26:28.832+01:00][DEBUG][plugins.fleet] Agents upgrade details telemetry: [{"target_version":"8.13.0-SNAPSHOT","state":"UPG_FAILED","error_msg":"failed download of agent binary: unable to download package: 3 errors occurred:\n\t* package '/Library/Elastic/Agent/data/elastic-agent-f383c6/downloads/elastic-agent-8.13.0-SNAPSHOT-darwin-aarch64.tar.gz' not found: open /Library/Elastic/Agent/data/elastic-agent-f383c6/downloads/elastic-agent-8.13.0-SNAPSHOT-darwin-aarch64.tar.gz: no such file or directory\n\t* call to 'https://artifacts.elastic.co/downloads/dummy/beats/elastic-agent/elastic-agent-8.13.0-SNAPSHOT-darwin-aarch64.tar.gz' returned unsuccessful status code: 404\n\t* call to 'https://artifacts.elastic.co/downloads/dummy/beats/elastic-agent/elastic-agent-8.13.0-SNAPSHOT-darwin-aarch64.tar.gz' returned unsuccessful status code: 404\n\n"}]

Checklist

Unit or functional tests were updated or added to match the most common scenarios

elasticmachine · 2023-12-14T10:37:27Z

Pinging @elastic/fleet (Team:Fleet)

apmmachine · 2023-12-14T10:37:34Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

/oblt-deploy : Deploy a Kibana instance using the Observability test environments.
/oblt-deploy-serverless : Deploy a serverless Kibana instance using the Observability test environments.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

juliaElastic · 2023-12-14T12:22:29Z

x-pack/plugins/fleet/server/collectors/agent_collectors.ts

+                },
+                {
+                  field: 'upgrade_details.metadata.error_msg.keyword',
+                  missing: '',


this ensures that documents are included where error_msg is missing, and uses empty string instead: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-multi-terms-aggregation.html#_missing_value_3

nchaulet · 2023-12-14T13:43:21Z

x-pack/plugins/fleet/server/collectors/agent_collectors.ts

@@ -135,6 +141,22 @@ export const getAgentData = async (
              ],
            },
          },
+          upgrade_details: {
+            multi_terms: {


@juliaElastic this will return only the first 10 items, it is what we want?

good catch! probably not, I'll increase this.

increased to 1000

x-pack/plugins/fleet/server/collectors/agent_collectors.ts

ycombinator · 2023-12-14T14:00:34Z

Implemented by doing a multi_terms aggregation to group the same target_version, state, error_msg values together.
Do we also want to include the agent count in each bucket in the telemetry event? @jlind23 @ycombinator

Yes, I think so because this would tell us how many Agents of a certain version failed upgrading at a certain step/state because of a certain error, so it will give us a fairly specific idea of where to focus any efforts. But I will defer to @jlind23 @cmacknz @nimarezainia to help decide what's useful in terms of telemetry.

jlind23 · 2023-12-14T14:07:50Z

Do we also want to include the agent count in each bucket in the telemetry event?

I would say yes, this will help us decide whether or not an issue is worth investigating based on the number of impacted Elastic Agent.

juliaElastic · 2023-12-14T14:41:38Z

Do we also want to include the agent count in each bucket in the telemetry event?

I would say yes, this will help us decide whether or not an issue is worth investigating based on the number of impacted Elastic Agent.

Thanks, added a agent_count field to capture how many agents have the same upgrade details.

nchaulet

code LGTM 🚀

kibana-ci · 2023-12-18T10:10:05Z

💚 Build Succeeded

Buildkite Build
Commit: acf8578

Metrics [docs]

✅ unchanged

History

💛 Build #183672 was flaky 9c84715
💛 Build #183598 was flaky 4681e83
💛 Build #183582 was flaky 8d34c49

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @juliaElastic

kibanamachine · 2023-12-18T10:14:13Z

💔 All backports failed

Status	Branch	Result
❌	8.12	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 173356

Questions ?

Please refer to the Backport tool documentation

@jlind23

Relates elastic#162448 Added upgrade details telemetry, publishing to `fleet-agents index` in telemetry cluster, each bucket as separate documents. Implemented by doing a `multi_terms` aggregation to group the same `target_version, state, error_msg` values together. Do we also want to include the agent count in each bucket in the telemetry event? @jlind23 @ycombinator Note: since this task runs every hour, it will most likely capture the `UPG_FAILED` states, since the other (success) states are temporarily on the agent docs, and removed if the upgrade is successful. E.g. 2 docs like the below become one telemetry event ``` // .fleet-agents upgrade_details: { target_version: '8.12.0', state: 'UPG_FAILED', metadata: { error_msg: 'Download failed', }, }, // telemetry event { target_version: '8.12.0', state: 'UPG_FAILED', error_msg: 'Download failed', } ``` To verify: - start kibana 8.13-SNAPSHOT locally - set an invalid agent download source in Fleet Settings - enroll an agent version 8.12-SNAPSHOT - upgrade to 8.13-SNAPSHOT with the API ``` POST kbn:/api/fleet/agents/<agent_id>/upgrade { "version": "8.13.0-SNAPSHOT", "force": true } ``` - wait 15m so that the upgrade goes to failed state - wait up to 1h for the telemetry task to run (speed up locally by setting a shorter interval in FleetUsageSender in kibana) - verify in debug logs: ``` [2023-12-14T14:26:28.832+01:00][DEBUG][plugins.fleet] Agents upgrade details telemetry: [{"target_version":"8.13.0-SNAPSHOT","state":"UPG_FAILED","error_msg":"failed download of agent binary: unable to download package: 3 errors occurred:\n\t* package '/Library/Elastic/Agent/data/elastic-agent-f383c6/downloads/elastic-agent-8.13.0-SNAPSHOT-darwin-aarch64.tar.gz' not found: open /Library/Elastic/Agent/data/elastic-agent-f383c6/downloads/elastic-agent-8.13.0-SNAPSHOT-darwin-aarch64.tar.gz: no such file or directory\n\t* call to 'https://artifacts.elastic.co/downloads/dummy/beats/elastic-agent/elastic-agent-8.13.0-SNAPSHOT-darwin-aarch64.tar.gz' returned unsuccessful status code: 404\n\t* call to 'https://artifacts.elastic.co/downloads/dummy/beats/elastic-agent/elastic-agent-8.13.0-SNAPSHOT-darwin-aarch64.tar.gz' returned unsuccessful status code: 404\n\n"}] ``` - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

Backport #173356 I got a merge conflict on backporting to 8.12, because the presets telemetry was not backportet, is that intentional? #172838

juliaElastic added 2 commits December 14, 2023 10:41

upgrade details telemetry

ec30a8f

added test coverage

8d34c49

juliaElastic added release_note:skip Skip the PR/issue when compiling release notes v8.12.0 labels Dec 14, 2023

juliaElastic self-assigned this Dec 14, 2023

juliaElastic requested a review from a team as a code owner December 14, 2023 10:37

botelastic bot added the Team:Fleet Team label for Observability Data Collection Fleet team label Dec 14, 2023

fix agg when error_msg missing

4681e83

juliaElastic commented Dec 14, 2023

View reviewed changes

nchaulet reviewed Dec 14, 2023

View reviewed changes

x-pack/plugins/fleet/server/collectors/agent_collectors.ts Show resolved Hide resolved

added higher size + count

7b31aab

juliaElastic and others added 2 commits December 14, 2023 15:44

renamed to agent_count

c86ac73

Merge branch 'main' into upgrade_details_telemetry

9c84715

nchaulet approved these changes Dec 14, 2023

View reviewed changes

Merge branch 'main' into upgrade_details_telemetry

acf8578

juliaElastic enabled auto-merge (squash) December 18, 2023 09:57

juliaElastic merged commit a61b864 into elastic:main Dec 18, 2023
39 checks passed

kibanamachine added the v8.13.0 label Dec 18, 2023

juliaElastic mentioned this pull request Dec 18, 2023

[8.12][Fleet] Upgrade details telemetry (#173356) #173502

Merged

juliaElastic added a commit that referenced this pull request Dec 18, 2023

[8.12][Fleet] Upgrade details telemetry (#173356) (#173502)

8904970

Backport #173356 I got a merge conflict on backporting to 8.12, because the presets telemetry was not backportet, is that intentional? #172838

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] Upgrade details telemetry #173356

[Fleet] Upgrade details telemetry #173356

juliaElastic commented Dec 14, 2023 •

edited by kibanamachine

Loading

elasticmachine commented Dec 14, 2023

apmmachine commented Dec 14, 2023

juliaElastic Dec 14, 2023

nchaulet Dec 14, 2023

juliaElastic Dec 14, 2023

juliaElastic Dec 14, 2023

ycombinator commented Dec 14, 2023 •

edited

Loading

jlind23 commented Dec 14, 2023

juliaElastic commented Dec 14, 2023 •

edited

Loading

nchaulet left a comment

kibana-ci commented Dec 18, 2023

kibanamachine commented Dec 18, 2023

[Fleet] Upgrade details telemetry #173356

[Fleet] Upgrade details telemetry #173356

Conversation

juliaElastic commented Dec 14, 2023 • edited by kibanamachine Loading

Summary

Checklist

elasticmachine commented Dec 14, 2023

apmmachine commented Dec 14, 2023

🤖 GitHub comments

juliaElastic Dec 14, 2023

Choose a reason for hiding this comment

nchaulet Dec 14, 2023

Choose a reason for hiding this comment

juliaElastic Dec 14, 2023

Choose a reason for hiding this comment

juliaElastic Dec 14, 2023

Choose a reason for hiding this comment

ycombinator commented Dec 14, 2023 • edited Loading

jlind23 commented Dec 14, 2023

juliaElastic commented Dec 14, 2023 • edited Loading

nchaulet left a comment

Choose a reason for hiding this comment

kibana-ci commented Dec 18, 2023

💚 Build Succeeded

Metrics [docs]

History

kibanamachine commented Dec 18, 2023

💔 All backports failed

Manual backport

Questions ?

juliaElastic commented Dec 14, 2023 •

edited by kibanamachine

Loading

ycombinator commented Dec 14, 2023 •

edited

Loading

juliaElastic commented Dec 14, 2023 •

edited

Loading