Agent goes to Offline status on reboot of Linux deb stand-alone agent enroll to Fleet

[ghost]

Kibana version:
Kibana: 7.10.0 BC5 cloud environment

Elasticsearch version:
Elasticsearch: 7.10.0 BC5 cloud environment

Host OS and Browser version:
Linux Debian 9 , All

Original install method (e.g. download page, yum, from source, etc.):
7.10.0 BC5 cloud environment from https://cloud.elastic.co

Description
Agent goes to Offline status on reboot of Linux deb stand-alone agent enroll to Fleet

Preconditions

1. 7.10.0 BC5 Kibana cloud environment(with commit 1796b5ec8fa1e60ccea63f2e5c25ccc665b92fdc) should be available.
2. Login into the above Kibana environment with 'elastic' superuser.
3. 7.10.0 BC5 Linux amd64.deb agent package should be downloaded and installed(on Linux Debian 9) from below location:
   https://staging.elastic.co/7.10.0-b55684ff/downloads/beats/elastic-agent/elastic-agent-7.10.0-amd64.deb

Steps to Reproduce

1. Download the "elastic-agent-7.10.0-amd64.deb" package.

2. SSH to the endpoint and place the agent package on /home/zeus location on Debian 9 Linux machine.

3. Hit the command to get the root/admin permission: sudo su root

4. Run the command: sudo dpkg -i elastic-agent-7.10.0-amd64.deb

5. Download elastic-agent.yml from Add agent ->Standalone mode on Agents tab in Kibana and add Kibana login username and password in outputs section of elastic-agent.yml (Modify ES_USERNAME and ES_PASSWORD in the outputs section of elastic-agent.yml to use your Elasticsearch credentials)
   elastic-agent_yml.zip

6. Copy elastic-agent.yml file to /home/zeus on Linux Debian 9 machine

7. Then execute: sudo cp /home/zeus/elastic-agent.yml /etc/elastic-agent/elastic-agent.yml

8. Run the following commands to run the standalone agent :-
   elastic-agent run

9. Execute the enroll command :-
   sudo elastic-agent enroll https://d9dc24b9e27749e8af99cb8d590ca97f.europe-west1.gcp.cloud.es.io:9243 Y1RKaHJIVUJHcXBIMWZsU1dLTEI6cEtfakhXVTJScUctUmJhUENiRFlZQQ==

10. Execute the below command to enable the agent :-
    systemctl enable elastic-agent

and observe that error is displayed in putty :-

Failed to reload daemon: The name org.freedesktop.PolicyKit1 was not provided by any .service files
update-rc.d: error: no runlevel symlinks to modify, aborting!

11. Execute the below command to start the agent :-
    systemctl start elastic-agent

and observe that following error is displayed in putty but agent comes in Online status and activity logs are displayed :-

Failed to start elastic-agent.service: The name org.freedesktop.PolicyKit1 was not provided by any .service files
See system logs and 'systemctl status elastic-agent.service' for details.

12. Run the command 'sudo reboot' and observe that agent goes to Offline status.

Test data
N/A

Impacted Test case id
https://elastic.testrail.io/index.php?/cases/view/34373

Actual Result
Agent goes to Offline status on reboot of Linux deb stand-alone agent enroll to Fleet

Expected Result
Agent should remain in Online status on reboot of Linux deb stand-alone agent enroll to Fleet

What's working

• Issue is not occurring on stand-alone agent enroll to Fleet with Linux rpm

What's not working

• N/A

Screenshot

1. Errors in putty on enabling and starting the agent:
   

2. Agent in Online status with activity logs on executing agent start command(before reboot):
   

3. Agent in Offline status after reboot:
   

4. Linux Debian 9 machine VSphere Template:
   

Logs
N/A

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[ghost]

@vipulsrivastava-qasource Please review the bug.

[ghost]

Reviewed and Assigned to @EricDavisX

[EricDavisX]

There is a lot to unpack in the steps there, but it boils down to a fairly simple case and is one we intend to support (and which works for .rpm usage as reported). I suspect it is due to the version of or relating files on that instance of Debian 9, as a first guess.

If we don't have it, I figure it would be good to confirm that the stand-alone Agent can be rebooted and that persistence is ok, before 'enrolling' the Agent which I think should largely be unrelated.

If we need to get specifics of file confirmations or logs, can you help confirm exactly where or which ones to pull please?

Also, @rahulgupta-qasource if you had a different instance of .deb usage to confirm against, so we can know if it is all .deb or just this OS version.

[EricDavisX]

I don't know if we want to adjust the impact label from high, until we know more. @ph fyi. @blakerouse if you have time to review.

[ph]

I am confused by this issue, If I understand it correctly the story starts with standalone mode and than switch to fleet?
Looking at the error this with systemctl, I think there is a problem with other services that prevent Elastic Agent to start?

[ph]

Like I've said on slack, lets have a look on the debian 9. But the error related to "freedesktop" makes me think that the os installation is broken.

[EricDavisX]

the above PR should fix this - we are expecting backports for 7.10.1 (not for 7.10.0 initial release).

[dikshachauhan-qasource]

Hi @EricDavisX ,

We have re-validated this issue on 8.0 snapshot kibana environment and found that issue is Not-Fixed and still reproducible.

Build details are as follows:
Agent hash: https://snapshots.elastic.co/8.0.0-432f821f/downloads/beats/elastic-agent/elastic-agent-8.0.0-SNAPSHOT-amd64.deb
BUILD 38085
COMMIT 228387cb6e3cdc35123f5a8bc40d3ad87d4f9e52
Linux Debian 9 machine VSphere Template used: atlqa>templates>qa-debian9-x64-endpoint-clone-base

Observation: While enrolling a standalone agent to fleet, we observed same issues mentioned at Steps 10-12 in defect description. Please refer below screenshot for details:

Hence, we are reopening this issue.

[dikshachauhan-qasource]

Hi @EricDavisX ,

We have performed the system reboot scenario on Linux deb standalone agent as per your request and below is our observation:

Agent hash: https://snapshots.elastic.co/8.0.0-432f821f/downloads/beats/elastic-agent/elastic-agent-8.0.0-SNAPSHOT-amd64.deb
BUILD 38085
COMMIT 228387cb6e3cdc35123f5a8bc40d3ad87d4f9e52

Steps followed:

• Download the " elastic-agent-8.0.0-SNAPSHOT-amd64.deb" package.
• SSH to the endpoint and place the agent package on /home/zeus location on Debian 9 Linux machine.
• Hit the command to get the root/admin permission: sudo su root
• Run the command: sudo dpkg -i elastic-agent-8.0.0-SNAPSHOT-amd64.deb
• Download elastic-agent.yml from Add agent ->Standalone mode on Agents tab in Kibana and add Kibana login username and password in outputs section of elastic-agent.yml (Modify ES_USERNAME and ES_PASSWORD in the outputs section of elastic-agent.yml to use your Elasticsearch credentials)
• Copy elastic-agent.yml file to /home/zeus on Linux Debian 9 machine
• Then execute: sudo cp /home/zeus/elastic-agent.yml /etc/elastic-agent/elastic-agent.yml
• Run the following commands to run the standalone agent :-
  elastic-agent run
• In kibana, observe data streaming started for standalone agent.
• Again, access another instance of Putty, and run the command 'sudo reboot' .
• Wait, for few minutes and again SSH to the endpoint.
• Refresh the data set page in Kibana.

Also, executed the below command to verify the status of agent :-
systemctl status elastic-agent
and observed response :-

elastic-agent.service - Agent manages other beats based on configuration provided.
   Loaded: loaded (/lib/systemd/system/elastic-agent.service; disabled; vendor preset: en
   Active: inactive (dead)
     Docs: https://www.elastic.co/products/beats/elastic-agent

Observation:

• On rebooting the endpoint, found elastic-agent as inactive.
• Hence, new data events were not generated as per the new timestamp on data set page in Kibana.

Screenshot:

[ph]

Fixed by #22593