[Self-Managed]: Unable to install secondary agent on self-managed kibana having Fleet Server running.

[amolnater-qasource]

Kibana version: 8.6 BC1 self-managed environment

Host OS and Browser version: Windows, All

Build details:

VERSION: 8.6.0 BC1 Kibana self-managed environment
BUILD: 58392
COMMIT: 50a7feb0a5eb068d3acccc49c83b9ccb6db6734f

Preconditions:

1. 8.6 BC1 self-managed environment should be available.
2. Fleet Server should be installed and a host url should be added.

Steps to reproduce:

1. Navigate to Agents tab.
2. Create Agent policy and run agent install command with --insecure.
3. Observe error:

Error: fail to enroll: fail to execute request to fleet-server: status code: 400, fleet-server returned an error: BadRequest, message: apikey auth response 4sUqioQB57iiFfqR0hGq: [401 Unauthorized] {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate with provided credentials and anonymous access is not allowed for this request","additional_unsuccessful_credentials":"API key: unable to find apikey with id 4sUqioQB57iiFfqR0hGq","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate with provided credentials and anonymous access is not allowed for this request","additional_unsuccessful_credentials":"API key: unable to find apikey with id 4sUqioQB57iiFfqR0hGq","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.6/fleet-troubleshooting.html
Error: enroll command failed with exit code: 1

Enrollment Token is correct and is not revoked for agent policy.
Screenshots:
Alerts:

Kibana details:

Expected Result:
User should be able to install secondary agent on self-managed kibana having Fleet Server running.

[amolnater-qasource]

@manishgupta-qasource Please review.

[manishgupta-qasource]

Secondary review for this ticket is Done

[cmacknz]

@joshdover or @michel-laterman any ideas on this one?

[joshdover]

Nothing obvious jumping out at me. @amolnater-qasource could you collect a couple things for the API key id that is returned in the error message:

• Results of GET _security/api_key?id=<id from error>
• Results of GET .fleet-enrollment-api-keys/_search?q=api_key_id:<id from error>

[amolnater-qasource]

Hi @joshdover

Thank you for looking into this.
We have revalidated this issue on latest 8.6 BC2 Kibana self-managed environment and found this issue reproducible.

Observations:

could you collect a couple things for the API key id that is returned in the error message:

• On the latest BC2, we are not getting API key error message.
• We are getting Error: enroll command failed with exit code: 1 even on installing agent with --insecure and without this flag.
• We are still not able to install secondary agent.
  Fleet-Server is running without any errors with Fleet Server integration.

Screenshots:

Build details:
BUILD: 58485
COMMIT: b52b34c2ff5216c395bd49c5fbc97744b646f34d

Fleet Server logs:
logs.zip

Please let us know if anything else is required from our end.
Thanks

[cmacknz]

The timeout here might be related to #1731, which is fixed in the latest snapshot but is not yet in the latest BC (BC2).

[amolnater-qasource]

Hi @cmacknz

Thank you for the update.
We will revalidate this on next 8.6 BC3 build as soon as it is available.

Thanks!

[cmacknz]

Following up, BC3 is available and BC4 will be available tomorrow.

[amolnater-qasource]

Hi @cmacknz

We have revalidated this issue on latest 8.6 BC4 and observed that we are able to install the secondary agent however it remains stuck in UPDATING state.

• For this we have reported a separate issue with all details and logs at [Self-Managed]: No data under data streams tab for secondary agent installed on self-managed kibana having Fleet Server running. #1861

Build details:

VERSION: 8.6.0 BC4 Kibana cloud environment
BUILD: 58612
COMMIT: 218162f282314db5b3833c84752dd24395949b3f

Hence we are closing this issue.

Thanks

[ghost]

Bug Conversion

• 01 new testcase is created under the 'Fleet Test Suite' at:
  • https://elastic.testrail.io/index.php?/cases/view/168586

Thanks