[Fleet]: Hosted fleet server not available on upgrading from 8.13.2>8.14.0.

[sukhwindersingh-qasource]

Kibana Build details:

VERSION: 8.13.2
BUILD: 72154
COMMIT: d4d06bdf0d1d5dcb4532f00d2cbaa83fc61bb877
Artifact Link: https://www.elastic.co/downloads/past-releases/elastic-agent-8-13-2

VERSION: 8.14.0
BUILD: 73520
COMMIT: c1513cd7e5a00eab209ba02d30cafd6945d75470

Preconditions:

• 8.13.2 Kibana cloud environment should be available.
• 3 policies should be present, with these 2 integrations - Defend , Osquery manager
• Agent should be installed on 3 or more endpoints.

Steps to reproduce:

• Navigate to Edit deployment.
• Now upgrade the 8.13.2 to 8.14.0
• Wait for the upgrade to be done.
• Navigate to the Fleets tab
• Observe Hosted fleet server not available on upgrading

Screen Recording:

Agents.-.Fleet.-.Elastic.Mozilla.Firefox.2024-04-22.15-53-16.mp4

Expected Result:

• Hosted fleet server should be available on upgrading from 8.13.2>8.14.0.

Whats Working:

• It is working when we upgrade the empty instance of 8.13.2 > 8.14.0.
• Issue got fixed after we force restart the integrations Server
  

[sukhwindersingh-qasource]

@amolnater-qasource Kindly review this
Thanks

[amolnater-qasource]

Secondary review for this ticket is Done.

[cmacknz]

Hmm, looks like Fleet server was reported as offline. Do you have the deployment ID so we can look up logs?

I tried this and it succeed so likely it is intermittent.

[sukhwindersingh-qasource]

Hi @cmacknz,

The deployment ID of the build is 9b9eb14524214a609c66172a4b05ca33. Please let us know if anything else is required.

Thanks!

[cmacknz]

Possibly this is the same as #3328

[juliaElastic]

This doesn't seem the same issue, I'm seeing policy docs with coordinator_idx: 1 way before the time of the upgrade:
2024-04-22T09:55:55.181Z

   {
        "_index": ".fleet-policies-7",
        "_id": "El0cBY8BUevs6eRkOcSy",
        "_score": null,
        "_source": {
          "coordinator_idx": 1,
          "policy_id": "policy-elastic-agent-on-cloud",
          "revision_idx": 5,
          "@timestamp": "2024-04-22T09:21:06.634Z"
        },

[juliaElastic]

I noticed that it's also slower for fleet-server to be healthy on upgrade/new deployment creation, the observability-perf tests consistently fail on this branch when expecting a healthy fleet-server in Functional Test: https://github.com/elastic/observability-perf/pull/771

I added a sleep of 2m, then fleet-server is healthy, tried with 1m, which wasn't enough.

Took a diagnostics from a cloud fleet-server where upgraded from 8.13.2 to 8.14.0. The fleet-server was in Updating state for a few minutes after the stack upgrade.
elastic-agent-diagnostics-2024-04-24T14-55-56Z-00.zip
Instance: https://admin.staging.foundit.no/deployments/ee8ec0f629cf4fc7940a0d5459910e97/activity

[kpollich]

I noticed that it's also slower for fleet-server to be healthy on upgrade/new deployment creation

I noticed this as well while executing our 8.14 test plan. The managed agent was initially updating for a few minutes before eventually going healthy. We should retry this on the next 8.14 BC with #3495 merged.

[juliaElastic]

Tested upgrading a cloud deployment from 8.13-SNAPSHOT to 8.14-SNAPSHOT and it was successful, and fleet-server became online quickly.
https://staging.found.no/deployments/661e87af059d4cca9a1114f8e4f6e950/activity

[kpollich]

Seems likely that this was fixed by this revert then, right? #3495

[juliaElastic]

Seems likely that this was fixed by this revert then, right? #3495

Yes, it seems so.

[kpollich]

Great! Thanks for testing. I'll close this. @amolnater-qasource @sukhwindersingh-qasource would you mind retesting? Feel free to reopen if there are further issues. Thank you!

[sukhwindersingh-qasource]

Hi @juliaElastic ,

We have validated this ticket on the latest 8.14.0 after upgrading it from 8.13.2 build and found the issue as FIXED. ✔️

Observations:

• Hosted fleet server now available on upgrading from 8.13.2>8.14.0.

Build Details:
VERSION: 8.14.0
BUILD: 73626
COMMIT: bcf6960778ae270d0894a8aab07f10197ee9b97f

Screenshot:

Hence, we are marking this issue as QA Validated.

Thanks!!