Improve `elastic-agent install` performance #3212

ycombinator · 2023-08-08T23:02:09Z

What does this PR do?

This PR makes elastic-agent install run slightly faster by concurrently copying files when the Agent is being installed on a host with SSDs. The same is done with Elastic Agent upgrades.

Summary of installation times

For all tests, the first installation result was discarded and the worse of the next two installation results (the result with the higher installation time) was used.

OS	Disk type	Before this PR (s)	After this PR (s)	Time savings (s)	Time savings (%)
MacOS	SSD	15.3s	13.9s	1.4s	9.2%
MacOS	HDD
Linux	SSD	12.2s	9.2s	3s	24.6%
Linux	HDD	45.7s	46s	-0.3s	-0.7%
Windows	SSD	9.6s	6.3s	3.3s	34.4%
Windows	HDD

MacOS - SSD

Tested on 2023 MacOS (Ventura 13.5) running SSDs.

Before this PR

$ time sudo ./elastic-agent install -n
Installing in non-interactive mode.Uninstalling current Elastic Agent... DONE
Copying files............................................................. DONE
Installing service... DONE
Starting service... DONE
Elastic Agent has been successfully installed.
sudo ./elastic-agent install -n  0.51s user 2.90s system 22% cpu 15.265 total
$ find . | wc -l
    3464
$ sudo find /Library/Elastic/Agent | wc -l
    3490

After this PR

$ time sudo ./elastic-agent install -n
Installing in non-interactive mode.
Uninstalling current Elastic Agent... DONE
Copying files................................................. DONE
Installing service... DONE
Starting service... DONE
Elastic Agent has been successfully installed.
sudo ./elastic-agent install -n  0.99s user 6.28s system 52% cpu 13.888 total

$ find . | wc -l
    3464
$ sudo find /Library/Elastic/Agent | wc -l
    3490

MacOS - HDD

TODO

Before this PR

After this PR

Linux - SSD

Tested on Google Cloud VM (Ubuntu 22) running SSDs (balanced persistent disk).

Before this PR

$ time sudo ./elastic-agent install -n
Installing in non-interactive mode.Uninstalling current Elastic Agent... DONE
Copying files................................................ DONE
Installing service..... DONE
Starting service... DONE
Elastic Agent has been successfully installed.

real	0m12.150s
user	0m0.003s
sys	0m0.008s

$ find . | wc -l
3448
$ sudo find /opt/Elastic/Agent/ | wc -l
3478

After this PR

$ time sudo ./elastic-agent install -n
Installing in non-interactive mode.
Uninstalling current Elastic Agent... DONE
Copying files...................................... DONE
Installing service.... DONE
Starting service.... DONE
Elastic Agent has been successfully installed.

real	0m9.148s
user	0m0.010s
sys	0m0.000s

$ find . | wc -l
3448
$ sudo find /opt/Elastic/Agent/ | wc -l
3478

Linux - HDD

Tested on Azure Cloud VM (Ubuntu 22) running HDDs (Standard HDD LRS).

Before this PR

$ time sudo ./elastic-agent install -n
Installing in non-interactive mode.Uninstalling current Elastic Agent... DONE
Copying files......................................................................................................................................................................... DONE
Installing service.... DONE
Starting service... DONE
Elastic Agent has been successfully installed.

real	0m45.694s
user	0m0.015s
sys	0m0.005s

$ find . | wc -l
3448
$ sudo find /opt/Elastic/Agent/ | wc -l
3478

After this PR

$ time sudo ./elastic-agent install -n
Installing in non-interactive mode.
Uninstalling current Elastic Agent... DONE
Copying files......................................................................................................................................................................... DONE
Installing service..... DONE
Starting service... DONE
Elastic Agent has been successfully installed.

real	0m45.957s
user	0m0.001s
sys	0m0.019s

$ find . | wc -l
3448
$ sudo find /opt/Elastic/Agent/ | wc -l
3478

Windows - SSD

Tested on 2022 Dell laptop (Windows 11 Pro) running SSDs.

Before this PR

PS>  Measure-Command { .\elastic-agent.exe install -n }


Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 9
Milliseconds      : 616
Ticks             : 96167912
TotalDays         : 0.000111305453703704
TotalHours        : 0.00267133088888889
TotalMinutes      : 0.160279853333333
TotalSeconds      : 9.6167912
TotalMilliseconds : 9616.7912

PS> (Get-ChildItem -Recurse | Measure-Object).Count
3454
PS> cd 'C:\Program Files\Elastic\Agent\'
PS> (Get-ChildItem -Recurse | Measure-Object).Count
3482

After this PR

PS> Measure-Command { .\elastic-agent.exe install -n }


Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 6
Milliseconds      : 316
Ticks             : 63167305
TotalDays         : 7.3110306712963E-05
TotalHours        : 0.00175464736111111
TotalMinutes      : 0.105278841666667
TotalSeconds      : 6.3167305
TotalMilliseconds : 6316.7305

PS> (Get-ChildItem -Recurse | Measure-Object).Count
3454
PS> cd 'C:\Program Files\Elastic\Agent\'
PS> (Get-ChildItem -Recurse | Measure-Object).Count
3482

Windows - HDD

TODO

Before this PR

After this PR

Why is it important?

Better user experience.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~I have made corresponding changes to the documentation~~
~~I have made corresponding change to the default configuration files~~
~~I have added tests that prove my fix is effective or that my feature works~~
I have added an entry in ./changelog/fragments using the changelog tool
~~I have added an integration test or an E2E test~~

mergify · 2023-08-08T23:02:45Z

This pull request does not have a backport label. Could you fix it @ycombinator? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 8./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

elasticmachine · 2023-08-08T23:08:22Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-10-04T21:37:50.734+0000
Duration: 26 min 36 sec

Test stats 🧪

Test	Results
Failed	0
Passed	6445
Skipped	59
Total	6504

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages.
run integration tests : Run the Elastic Agent Integration tests.
run end-to-end tests : Generate the packages and run the E2E Tests.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

mergify · 2023-08-11T13:01:51Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b concurrent-install upstream/concurrent-install
git merge upstream/main
git push upstream concurrent-install

elasticmachine · 2023-09-01T18:51:31Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	98.78% (`81/82`)	👍
Files	66.0% (`198/300`)	👍 0.333
Classes	65.171% (`363/557`)	👍 0.18
Methods	52.333% (`1144/2186`)	👍 0.022
Lines	37.982% (`13027/34298`)	👎 -0.023
Conditionals	100.0% (`0/0`)	💚

elasticmachine · 2023-09-03T12:50:36Z

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

faec

Looks good, modulo the question about comments

faec · 2023-09-06T21:24:15Z

internal/pkg/agent/install/install.go

+func HasAllSSDs() bool {
+	block, err := ghw.Block()
+	if err != nil {
+		//fmt.Fprintf(os.Stdout, "HasAllSSDs: ghw.Block() returned error: %s\n", err.Error())


Are the commented Fprintf's left here intentionally or just leftover from testing?

Leftovers, will remove. Thanks!

mergify · 2023-09-08T12:45:04Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b concurrent-install upstream/concurrent-install
git merge upstream/main
git push upstream concurrent-install

ycombinator · 2023-09-11T13:49:08Z

Currently waiting on otiai10/copy#123 to be merged, then will update go.mod on this PR.

mergify · 2023-09-21T16:28:52Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b concurrent-install upstream/concurrent-install
git merge upstream/main
git push upstream concurrent-install

ycombinator · 2023-09-29T18:54:25Z

Hey @michalpristas, I was wondering if you have a Windows machine with an HDD (as opposed to SSD). @pierrehilbert suggested that you might.

If you do, would you mind testing the performance of elastic-agent install on that machine before and after this PR? You can follow the same steps that I did for Windows - SSD (scroll down a bit).

If you don't have such a machine, just let me know. Thanks either way!

ycombinator · 2023-10-02T18:18:54Z

Hey @michalpristas, I was wondering if you have a Windows machine with an HDD (as opposed to SSD). @pierrehilbert suggested that you might.

If you do, would you mind testing the performance of elastic-agent install on that machine before and after this PR? You can follow the same steps that I did for Windows - SSD (scroll down a bit).

If you don't have such a machine, just let me know. Thanks either way!

Chatted with @michalpristas off-PR. He doesn't have an HDD machine where elastic-agent can be installed.

The HDD test is mainly to make sure there's no regression in performance with this change on HDDs, which I tested for Linux so I think that's good enough.

…f dependency has been released

elastic-sonarqube · 2023-10-04T23:47:50Z

SonarQube Quality Gate

0 Bugs
0 Vulnerabilities
0 Security Hotspots
1 Code Smell

59.1% Coverage
0.0% Duplication

ycombinator · 2023-10-05T01:03:57Z

@cmacknz Worth backporting to 8.11?

cmacknz · 2023-10-24T18:14:42Z

Missed the original ping here, not a bug fix so we shouldn't have backported.

ycombinator · 2023-10-31T00:42:11Z

Missed the original ping here, not a bug fix so we shouldn't have backported.

👍 This PR was not backported so we're good. Thanks!

mergify bot assigned ycombinator Aug 8, 2023

mergify bot added the backport-skip label Aug 8, 2023

ycombinator changed the title ~~Concurrent install~~ Improve elastic-agent install performance Aug 9, 2023

ycombinator force-pushed the concurrent-install branch from 44b8691 to ed01563 Compare August 28, 2023 13:44

ycombinator mentioned this pull request Aug 29, 2023

Make copying files concurrent otiai10/copy#120

Closed

ycombinator force-pushed the concurrent-install branch 4 times, most recently from 79342af to 6b04bf4 Compare September 1, 2023 18:21

ycombinator added the enhancement New feature or request label Sep 1, 2023

ycombinator marked this pull request as ready for review September 1, 2023 18:25

ycombinator requested a review from a team as a code owner September 1, 2023 18:25

ycombinator requested review from michalpristas and faec September 1, 2023 18:25

pierrehilbert added the Team:Elastic-Agent Label for the Agent team label Sep 3, 2023

faec approved these changes Sep 6, 2023

View reviewed changes

ycombinator force-pushed the concurrent-install branch from 0fb47e2 to 0cdbda4 Compare September 8, 2023 23:22

ycombinator mentioned this pull request Sep 9, 2023

Feature/concurrent dcopy otiai10/copy#123

Merged

ycombinator force-pushed the concurrent-install branch from 2bd0ff0 to 343ad8f Compare September 29, 2023 18:53

ycombinator force-pushed the concurrent-install branch from 343ad8f to af1c8a1 Compare October 2, 2023 18:00

ycombinator added 22 commits October 4, 2023 14:37

Update deps

027b7dc

Updating deps

11ab16e

Updating deps

d5a4cf0

Updating dep

2b36650

Update setting value

a87dd40

Detect SSDs

6155a28

Updating deps

13f1472

Updating deps

50d685a

Use in upgrade

a8ec7ec

Comment out debug logging statements

52877a6

Updating NOTICE.txt

b976d0a

Adding replace exception

44b04af

Adding CHANGELOG entry

c229483

Remove leftover commented out logging statements

bd68606

Update copy module branch

85c887d

Update PR reference

a2e289f

Updating dependency to released version

97c8b1c

Remove exception from replace-allow-list now that new minor version o…

75955cb

…f dependency has been released

Ran make notice

513b406

Making HasAllSSDs testable

87b75f2

Removing debugging statements

9668731

Adding unit tests

84cac6a

ycombinator force-pushed the concurrent-install branch from b88b270 to 84cac6a Compare October 4, 2023 21:37

ycombinator merged commit a4003e0 into elastic:main Oct 5, 2023
11 checks passed

ycombinator deleted the concurrent-install branch October 5, 2023 04:39

cmacknz mentioned this pull request Oct 24, 2023

Failing to detect SSDs in copyDir should not be a fatal error. #3653

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve `elastic-agent install` performance #3212

Improve `elastic-agent install` performance #3212

ycombinator commented Aug 8, 2023 •

edited

Loading

mergify bot commented Aug 8, 2023

elasticmachine commented Aug 8, 2023 •

edited

Loading

Build stats

Test stats 🧪

mergify bot commented Aug 11, 2023

elasticmachine commented Sep 1, 2023 •

edited

Loading

elasticmachine commented Sep 3, 2023

faec left a comment

faec Sep 6, 2023

ycombinator Sep 7, 2023

mergify bot commented Sep 8, 2023

ycombinator commented Sep 11, 2023

mergify bot commented Sep 21, 2023

ycombinator commented Sep 29, 2023 •

edited

Loading

ycombinator commented Oct 2, 2023 •

edited

Loading

elastic-sonarqube bot commented Oct 4, 2023

ycombinator commented Oct 5, 2023

cmacknz commented Oct 24, 2023

ycombinator commented Oct 31, 2023

Improve elastic-agent install performance #3212

Improve elastic-agent install performance #3212

Conversation

ycombinator commented Aug 8, 2023 • edited Loading

What does this PR do?

Summary of installation times

MacOS - SSD

Before this PR

After this PR

MacOS - HDD

Before this PR

After this PR

Linux - SSD

Before this PR

After this PR

Linux - HDD

Before this PR

After this PR

Windows - SSD

Before this PR

After this PR

Windows - HDD

Before this PR

After this PR

Why is it important?

Checklist

mergify bot commented Aug 8, 2023

elasticmachine commented Aug 8, 2023 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

mergify bot commented Aug 11, 2023

elasticmachine commented Sep 1, 2023 • edited Loading

🌐 Coverage report

elasticmachine commented Sep 3, 2023

faec left a comment

Choose a reason for hiding this comment

faec Sep 6, 2023

Choose a reason for hiding this comment

ycombinator Sep 7, 2023

Choose a reason for hiding this comment

mergify bot commented Sep 8, 2023

ycombinator commented Sep 11, 2023

mergify bot commented Sep 21, 2023

ycombinator commented Sep 29, 2023 • edited Loading

ycombinator commented Oct 2, 2023 • edited Loading

elastic-sonarqube bot commented Oct 4, 2023

ycombinator commented Oct 5, 2023

cmacknz commented Oct 24, 2023

ycombinator commented Oct 31, 2023

Improve `elastic-agent install` performance #3212

Improve `elastic-agent install` performance #3212

ycombinator commented Aug 8, 2023 •

edited

Loading

elasticmachine commented Aug 8, 2023 •

edited

Loading

elasticmachine commented Sep 1, 2023 •

edited

Loading

ycombinator commented Sep 29, 2023 •

edited

Loading

ycombinator commented Oct 2, 2023 •

edited

Loading