Alter agent install to detect installation from package #30289

michel-laterman · 2022-02-08T21:16:27Z

What does this PR do?

Alter the elastic-agent install command to not run the installation or
uninstall steps (on failure) when the agent has been installed via the
package manager of a Linux os. Instead the enroll command will be
executed.

Why is it important?

The current documentation/Kibana gives install commands, having these fail can be frustrating.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
[] ~~I have made corresponding changes to the documentation~~
I have made corresponding change to the default configuration files PR
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Install the rpm/deb package and run the install command that Kibana provides:

vagrant@ubuntu-focal:~$ sudo dpkg -i elastic-agent-8.2.0-SNAPSHOT-amd64.deb
Selecting previously unselected package elastic-agent.
(Reading database ... 69907 files and directories currently installed.)
Preparing to unpack elastic-agent-8.2.0-SNAPSHOT-amd64.deb ...
Unpacking elastic-agent (8.2.0) ...
Setting up elastic-agent (8.2.0) ...
Processing triggers for systemd (245.4-4ubuntu3.15) ...
vagrant@ubuntu-focal:~$ sudo elastic-agent install --url=https://579a0b58293d477da089b3e048311a63.fleet.us-central1.gcp.qa.cld.elstc.co:443 --enrollment-token=NTFqWF8zNEIyWVJ5Qm1EbkY1dzg6RUZVd1lIVEVTQk9oeXVtY3FHZ2pnZw==
Installed as a system package, installation will not be altered.
{"log.level":"info","@timestamp":"2022-02-16T00:10:23.061Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":455},"message":"Starting enrollment to URL: https://579a0b58293d477da089b3e048311a63.fleet.us-central1.gcp.qa.cld.elstc.co:443/","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-16T00:10:24.414Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":253},"message":"Elastic Agent might not be running; unable to trigger restart","ecs.version":"1.6.0"}
Successfully enrolled the Elastic Agent.
Elastic Agent has been successfully installed.
vagrant@ubuntu-focal:~$ ls /opt/
vagrant@ubuntu-focal:~$ systemctl status elastic-agent.service
● elastic-agent.service - Agent manages other beats based on configuration provided.
     Loaded: loaded (/lib/systemd/system/elastic-agent.service; disabled; vendor preset: enabled)
     Active: inactive (dead)
       Docs: https://www.elastic.co/beats/elastic-agent

Related issues

Closes [Agent] Using Fleet Server with 'install' command with RPM/DEB is not supported, can we fix the install cmd to check if RPM/DEB is installed and skip to the 'enroll' step? #25967

Alter the elastic-agent install command to not run the installation or uninstall steps (on failure) when the agent has been installed via the package manager of a Linux os. Instead the enroll command will be executed.

mergify · 2022-02-08T21:17:42Z

This pull request does not have a backport label. Could you fix it @michel-laterman? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

michel-laterman · 2022-02-08T21:19:53Z

Vagrantfile

+  {:name => "opensuse153", :box => "bento/opensuse-leap-15.3", :platform => "opensuse"},
+  {:name => "sles12", :box => "elastic/sles-12-x86_64", :platform => "sles"},


opensuse/sles12 added to help test suse

michel-laterman · 2022-02-08T21:21:15Z

x-pack/elastic-agent/pkg/agent/cmd/install.go

-		err = install.StartService()
+	cfgFile := paths.ConfigFile()
+	if status != install.PackageInstall {
+		err = install.Install(cfgFile) // can't run install for package installed agent, but must stop the running service and re-enroll, then start the service?


not sure if we need to do the commented steps, currently the enroll command will be ran with the --from-install flag, but I don't know if that's correct

Not sure either, what is the behavior in the OS?

Do not need to stop the running service, it should just enroll and then trigger a restart through the Elastic Agent control protocol. It should not need to be restarted through the service manager of the host.

michel-laterman · 2022-02-08T21:21:51Z

x-pack/elastic-agent/pkg/agent/install/install_unix.go

+	// NOTE searching for english words might not be a great idea as far as portability goes.
+	// list all installed packages then search for paths.BinaryName?
+	// dpkg is strange as the remove and purge processes leads to the package bing isted after a remove, but not after a purge


would like feedback for this suggestion

This is expected because remove does not remove modified configuration files, those will remain. Only remove with purge completely removes all data on the system related to a package.

Sadly yes, this is expected.

i'll keep the comments here as the extra documentation can help if anyone else needs to work on this

elasticmachine · 2022-02-09T01:39:50Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-02-24T16:27:19.739+0000
Duration: 158 min 55 sec

Test stats 🧪

Test	Results
Failed	0
Passed	38335
Skipped	3320
Total	41655

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

…tion

elasticmachine · 2022-02-16T00:27:52Z

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

michel-laterman · 2022-02-16T00:28:00Z

/test

blakerouse · 2022-02-16T02:01:46Z

x-pack/elastic-agent/pkg/agent/install/install_unix.go

+	// If the package has been installed, the status starts with "install"
+	// If the package has been removed (but not pruged) status starts with "deinstall"
+	// If purged or never installed, rc is 1
+	if _, err := os.Stat("/etc/dpkg"); err == nil {


What do you thing about doing a which dpkg-query instead? Only thing would be is which always present on a system? I believe so.

blakerouse · 2022-02-16T02:20:12Z

x-pack/elastic-agent/pkg/agent/install/install_unix.go

+	// NOTE searching for english words might not be a great idea as far as portability goes.
+	// list all installed packages then search for paths.BinaryName?
+	// dpkg is strange as the remove and purge processes leads to the package bing isted after a remove, but not after a purge


This is expected because remove does not remove modified configuration files, those will remain. Only remove with purge completely removes all data on the system related to a package.

blakerouse · 2022-02-16T02:22:57Z

x-pack/elastic-agent/pkg/agent/install/install_unix.go

+	// If the package has been removed (but not pruged) status starts with "deinstall"
+	// If purged or never installed, rc is 1
+	if _, err := os.Stat("/etc/dpkg"); err == nil {
+		out, err := exec.Command("dpkg-query", "-W", "-f", "${Status}", paths.BinaryName).Output()


It would be better to use the package name explicitly instead of the binary name. Even if they are the same at the moment use a specific constant for that would be better, that way if it ever changes.

Could you also use -s which will only show the status?

The output of -s includes more information than what we need:

vagrant@ubuntu-focal:~$ dpkg-query -s elastic-agent Package: elastic-agent Status: install ok installed Priority: extra Section: default Installed-Size: 427208 Maintainer: <@ace0c0ea54fb> Architecture: amd64 Version: 8.2.0 Conffiles: /etc/elastic-agent/.elastic-agent.active.commit 75bbbe44ef52356bb5e85a458166b1b6 /etc/elastic-agent/elastic-agent.reference.yml 23aa69e9e641a43d7daaa0975edf0ec6 /etc/elastic-agent/elastic-agent.yml f169e731b8e4c8a571dca0107954534d /etc/init.d/elastic-agent 79b301b56cc62629b9d276d6ae24ca2f Description: Agent manages other beats based on configuration provided. License: Elastic-License Vendor: Elastic Homepage: https://www.elastic.co/beats/elastic-agent

where the -W -f '${Status}' flags will give just that Status line

@blakerouse, do you know where the package name is defined?

I believe it is using the name of the beat for the package name, so the way you have it works. I am just suggesting we use a different constant, with the same value. Just to future proof it that maybe one day they could be different.

Its more of a nitpick and if you prefer they way you have it, that is okay as well.

One question, here, I see we are interacting directly with the CLI, is there any C-Libary of even go package that we can use to abstract the package manager?

I'm not finding a c or go library that can query dpkg/rpm for installed packages

After a bit of digging we can find that the package name is defined by our Mage files as the BeatServiceName, this defaults to the same as BeatName for almost all cases (exception being heartbeat).

beats/dev-tools/mage/settings.go

Lines 70 to 71 in 2379370

BeatName = EnvOr("BEAT_NAME", filepath.Base(CWD()))

BeatServiceName = EnvOr("BEAT_SERVICE_NAME", BeatName)

The value is set to the directory name we run mage out of (i.e., x-pack/elastic-agent). So I don't think it's a good idea to introduce a new variable for package name within the current codebase as it won't be picked up by our tooling.

blakerouse · 2022-02-16T02:23:19Z

x-pack/elastic-agent/pkg/agent/install/install_unix.go

+	// check rhel and sles based systems (or systems that use rpm)
+	// if package has been installed query retuns with a list of associated files.
+	// otherwise if uninstalled, or has never been installled status ends with "not installed"
+	if _, err := os.Stat("/etc/rpm"); err == nil {


What about which rpm here as well?

…tion

michel-laterman · 2022-02-17T22:50:16Z

/test

Vagrantfile

ph · 2022-02-22T19:12:23Z

x-pack/elastic-agent/pkg/agent/cmd/install.go

-		err = install.StartService()
+	cfgFile := paths.ConfigFile()
+	if status != install.PackageInstall {
+		err = install.Install(cfgFile) // can't run install for package installed agent, but must stop the running service and re-enroll, then start the service?


Not sure either, what is the behavior in the OS?

ph · 2022-02-22T19:15:59Z

x-pack/elastic-agent/pkg/agent/install/install_unix.go

+	// NOTE searching for english words might not be a great idea as far as portability goes.
+	// list all installed packages then search for paths.BinaryName?
+	// dpkg is strange as the remove and purge processes leads to the package bing isted after a remove, but not after a purge


Sadly yes, this is expected.

ph · 2022-02-22T19:18:42Z

x-pack/elastic-agent/pkg/agent/install/install_unix.go

+	// If the package has been removed (but not pruged) status starts with "deinstall"
+	// If purged or never installed, rc is 1
+	if _, err := os.Stat("/etc/dpkg"); err == nil {
+		out, err := exec.Command("dpkg-query", "-W", "-f", "${Status}", paths.BinaryName).Output()


One question, here, I see we are interacting directly with the CLI, is there any C-Libary of even go package that we can use to abstract the package manager?

Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com>

blakerouse

Looks ready to me!

ph

LGTM thanks @michel-laterman

Alter agent install to detect installation from package

d2924aa

Alter the elastic-agent install command to not run the installation or uninstall steps (on failure) when the agent has been installed via the package manager of a Linux os. Instead the enroll command will be executed.

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Feb 8, 2022

michel-laterman added enhancement Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team labels Feb 8, 2022

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Feb 8, 2022

mergify bot assigned michel-laterman Feb 8, 2022

mergify bot added the backport-skip Skip notification from the automated backport with mergify label Feb 8, 2022

michel-laterman commented Feb 8, 2022

View reviewed changes

Merge remote-tracking branch 'origin/main' into package-install-detec…

1122db3

…tion

michel-laterman marked this pull request as ready for review February 16, 2022 00:27

blakerouse reviewed Feb 16, 2022

View reviewed changes

michel-laterman added 2 commits February 16, 2022 09:21

Merge remote-tracking branch 'origin/main' into package-install-detec…

cd28a37

…tion

Check package manager by running which instead of looking for file

d381fcf

ph self-requested a review February 21, 2022 13:46

ph suggested changes Feb 22, 2022

View reviewed changes

michel-laterman and others added 2 commits February 23, 2022 12:49

Update Vagrantfile

cd36f10

Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com>

Remove commented question

86cf7ef

blakerouse approved these changes Feb 24, 2022

View reviewed changes

ph approved these changes Feb 24, 2022

View reviewed changes

michel-laterman mentioned this pull request Feb 24, 2022

Update install command for rpm/deb detection elastic/observability-docs#1616

Merged

michel-laterman merged commit bc84d77 into elastic:main Feb 24, 2022

michel-laterman deleted the package-install-detection branch February 24, 2022 20:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Alter agent install to detect installation from package #30289

Alter agent install to detect installation from package #30289

michel-laterman commented Feb 8, 2022 •

edited

Loading

mergify bot commented Feb 8, 2022

michel-laterman Feb 8, 2022

michel-laterman Feb 8, 2022

ph Feb 22, 2022

blakerouse Feb 23, 2022

michel-laterman Feb 8, 2022

blakerouse Feb 16, 2022

ph Feb 22, 2022

michel-laterman Feb 24, 2022

elasticmachine commented Feb 9, 2022 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

elasticmachine commented Feb 16, 2022

michel-laterman commented Feb 16, 2022

blakerouse Feb 16, 2022

blakerouse Feb 16, 2022

blakerouse Feb 16, 2022

michel-laterman Feb 16, 2022

michel-laterman Feb 16, 2022

blakerouse Feb 17, 2022

ph Feb 22, 2022

michel-laterman Feb 23, 2022

michel-laterman Feb 23, 2022

ph Feb 24, 2022

blakerouse Feb 16, 2022

michel-laterman commented Feb 17, 2022

ph Feb 22, 2022

ph Feb 22, 2022

ph Feb 22, 2022

blakerouse left a comment

ph left a comment

		{:name => "opensuse153", :box => "bento/opensuse-leap-15.3", :platform => "opensuse"},
		{:name => "sles12", :box => "elastic/sles-12-x86_64", :platform => "sles"},

	BeatName = EnvOr("BEAT_NAME", filepath.Base(CWD()))
	BeatServiceName = EnvOr("BEAT_SERVICE_NAME", BeatName)

Alter agent install to detect installation from package #30289

Alter agent install to detect installation from package #30289

Conversation

michel-laterman commented Feb 8, 2022 • edited Loading

What does this PR do?

Why is it important?

Checklist

How to test this PR locally

Related issues

mergify bot commented Feb 8, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Feb 9, 2022 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

elasticmachine commented Feb 16, 2022

michel-laterman commented Feb 16, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michel-laterman commented Feb 17, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

blakerouse left a comment

Choose a reason for hiding this comment

ph left a comment

Choose a reason for hiding this comment

michel-laterman commented Feb 8, 2022 •

edited

Loading

elasticmachine commented Feb 9, 2022 •

edited by jenkins-beats-ci bot

Loading