Change local fleet-server connection to localhost:8221 #1867

michel-laterman · 2022-12-01T21:47:49Z

What does this PR do?

Fix an issue where the local fleet-server port was not properly used by the elastic-agent when running an instance of fleet-server.

Why is it important?

When running a large scale of agents, the fleet-server may hit the connection limit if it's running on the same port as the other agents. this causes the elastic-agent running the fleet-server to be marked as degraded.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Fix an issue where the local fleet-server port was not properly used by the elastic-agent when running an instance of fleet-server.

michel-laterman · 2022-12-01T21:52:07Z

internal/pkg/agent/cmd/enroll_cmd.go

+	if c.options.FleetServer.InternalPort == 0 {
+		c.options.FleetServer.InternalPort = defaultFleetServerInternalPort
+	}


The InternalURL is set when the agent is creating the fleet tls settings (line 321), however it expects a non-zero internal port value in order to do so.

this is fine. we have a check like this inside createFleetServerBootstrapConfig i'm ok with moving this up.

elasticmachine · 2022-12-01T22:01:55Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-12-06T18:04:24.233+0000
Duration: 17 min 6 sec

Test stats 🧪

Test	Results
Failed	0
Passed	4661
Skipped	13
Total	4674

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages.
run integration tests : Run the Elastic Agent Integration tests.
run end-to-end tests : Generate the packages and run the E2E Tests.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2022-12-01T23:11:41Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	98.333% (`59/60`)	👍
Files	69.082% (`143/207`)	👍
Classes	69.133% (`271/392`)	👍
Methods	53.988% (`819/1517`)	👍
Lines	39.154% (`8868/22649`)	👍 0.016
Conditionals	100.0% (`0/0`)	💚

internal/pkg/agent/cmd/enroll_cmd.go

michalpristas

have you tested this with 2 agents
one running fleet server, other one enrolling with this fleet server.
please make sure this scenario works

michel-laterman · 2022-12-06T00:57:04Z

I've tested what port is used by adding a local debug line in remote/client.go

When I enroll a new agent running the fleet-server I see that the enroll request targets 8220

{"log.level":"debug","@timestamp":"2022-12-05T16:25:47.701-0800","log.origin":{"file.name":"remote/client.go","file.line":177},"message":"REMOTE DEBUG request URL https://Michels-MacBook-Pro.local:8220/api/fleet/agents/enroll?","ecs.version":"1.6.0"}

However, when the agent is running afterwards it switches to 8221:

{"log.level":"debug","@timestamp":"2022-12-06T00:49:28.169Z","log.origin":{"file.name":"remote/client.go","file.line":177},"message":"REMOTE DEBUG request URL https://localhost:8221/api/fleet/agents/30295209-b53d-4aba-9c40-cf8c3ef38091/checkin?","ecs.version":"1.6.0"}

Enrolling another agent always targets 8221:

{"log.level":"debug","@timestamp":"2022-12-06T00:52:55.565Z","log.origin":{"file.name":"remote/client.go","file.line":177},"message":"REMOTE DEBUG request URL https://192.168.4.20:8220/api/fleet/agents/a29b0ae9-f172-48da-bc69-a0b650fb3e08/checkin?","ecs.version":"1.6.0"}

michalpristas · 2022-12-06T14:09:27Z

However, when the agent is running afterwards it switches to 8221:

is this for remote agent? remote agent should not call this port just the one running fleet-server.

michel-laterman · 2022-12-06T16:29:45Z

Oops, sorry I was not clear;

The local agent does the swap from 8220 to 8221 when enrolling/running.
A remote agent always uses 8220

michalpristas · 2022-12-06T17:04:11Z

just to clarify initial implementation and why we have this.
first let's distinguish internal and external communication we have here. internal is agent which is running this particular fleet server, external are any other enrolled agents
when we use single port for both internal and external communication, it can happen that agent which is running fleet server is throtled on TooManyRequests because it shares the resources with all other agents. In rare occasions this can lead to agent running fleet server being forcefully unenrolled because multiple checkins were dropped and Unenrollment timeout has passed. When this happens we lose all the agents enrolled to this particular fleet server.

So we introduced internal port which is used only by agent running fleet server. this prevents throttling and force unenrollment of all agents due to heavy load.

sonarcloud · 2022-12-06T18:05:39Z

SonarCloud Quality Gate failed.

0 Bugs
0 Vulnerabilities
0 Security Hotspots
1 Code Smell

No Coverage information
7.6% Duplication

michel-laterman · 2022-12-06T19:55:51Z

After a brief discussion with @michalpristas we have decided to keep the behaviour of the local agent that this PR introduces: using 8220 to enrol and 8221 when running normally as it should not run into load issues during the bootstrap process.
If required we may be able to change the enrol port for the bootstrap process in the future

jlind23 · 2022-12-08T08:51:59Z

@michel-laterman can we proceed with this change or is there anything else missing?

* Change local fleet-server connection to localhost:8221 Fix an issue where the local fleet-server port was not properly used by the elastic-agent when running an instance of fleet-server. * Fix typo * Add additional debug line in remote client * change to certificate verfication for local port (cherry picked from commit 8c7537b) # Conflicts: # internal/pkg/agent/cmd/enroll_cmd.go

* Change local fleet-server connection to localhost:8221 Fix an issue where the local fleet-server port was not properly used by the elastic-agent when running an instance of fleet-server. * Fix typo * Add additional debug line in remote client * change to certificate verfication for local port (cherry picked from commit 8c7537b)

* Change local fleet-server connection to localhost:8221 Fix an issue where the local fleet-server port was not properly used by the elastic-agent when running an instance of fleet-server. * Fix typo * Add additional debug line in remote client * change to certificate verfication for local port (cherry picked from commit 8c7537b) Co-authored-by: Michel Laterman <82832767+michel-laterman@users.noreply.github.com>

Change local fleet-server connection to localhost:8221

800f0d0

Fix an issue where the local fleet-server port was not properly used by the elastic-agent when running an instance of fleet-server.

michel-laterman added bug Something isn't working Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team backport-v8.5.0 Automated backport with mergify backport-v8.6.0 Automated backport with mergify labels Dec 1, 2022

michel-laterman requested a review from michalpristas December 1, 2022 21:47

mergify bot assigned michel-laterman Dec 1, 2022

michel-laterman commented Dec 1, 2022

View reviewed changes

Fix typo

26bf714

michel-laterman marked this pull request as ready for review December 2, 2022 00:49

michel-laterman requested a review from a team as a code owner December 2, 2022 00:49

michel-laterman requested review from aleksmaus and removed request for a team December 2, 2022 00:49

michel-laterman commented Dec 2, 2022

View reviewed changes

internal/pkg/agent/cmd/enroll_cmd.go Outdated Show resolved Hide resolved

joshdover reviewed Dec 2, 2022

View reviewed changes

internal/pkg/agent/cmd/enroll_cmd.go Outdated Show resolved Hide resolved

cmacknz requested a review from blakerouse December 2, 2022 14:41

michalpristas reviewed Dec 5, 2022

View reviewed changes

Add additional debug line in remote client

8cd8d3a

change to certificate verfication for local port

e5604e8

michalpristas approved these changes Dec 7, 2022

View reviewed changes

joshdover approved these changes Dec 7, 2022

View reviewed changes

michel-laterman merged commit 8c7537b into elastic:main Dec 8, 2022

michel-laterman deleted the fleet-port branch December 8, 2022 17:21

mergify bot mentioned this pull request Dec 8, 2022

[8.5](backport #1867) Change local fleet-server connection to localhost:8221 #1915

Closed

mergify bot mentioned this pull request Dec 8, 2022

[8.6](backport #1867) Change local fleet-server connection to localhost:8221 #1916

Merged

naemono mentioned this pull request Dec 8, 2022

Fix Elastic Agent Fleet Server without TLS e2e tests elastic/cloud-on-k8s#6214

Merged

cmacknz mentioned this pull request Jan 19, 2023

Fleet Server listens on loopback when TLS is disabled in v8.6.0 #2197

Closed

thbkrkr mentioned this pull request Jan 31, 2023

TestFleetAgentWithoutTLS is failing elastic/cloud-on-k8s#6367

Closed

michel-laterman mentioned this pull request Apr 9, 2024

Backport elastic-agent#1867 elastic/beats#38785

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Change local fleet-server connection to localhost:8221 #1867

Change local fleet-server connection to localhost:8221 #1867

michel-laterman commented Dec 1, 2022

michel-laterman Dec 1, 2022

michalpristas Dec 5, 2022

elasticmachine commented Dec 1, 2022 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Dec 1, 2022 •

edited

Loading

michalpristas left a comment

michel-laterman commented Dec 6, 2022

michalpristas commented Dec 6, 2022

michel-laterman commented Dec 6, 2022

michalpristas commented Dec 6, 2022

sonarcloud bot commented Dec 6, 2022

michel-laterman commented Dec 6, 2022

jlind23 commented Dec 8, 2022

Change local fleet-server connection to localhost:8221 #1867

Change local fleet-server connection to localhost:8221 #1867

Conversation

michel-laterman commented Dec 1, 2022

What does this PR do?

Why is it important?

Checklist

michel-laterman Dec 1, 2022

Choose a reason for hiding this comment

michalpristas Dec 5, 2022

Choose a reason for hiding this comment

elasticmachine commented Dec 1, 2022 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

elasticmachine commented Dec 1, 2022 • edited Loading

🌐 Coverage report

michalpristas left a comment

Choose a reason for hiding this comment

michel-laterman commented Dec 6, 2022

michalpristas commented Dec 6, 2022

michel-laterman commented Dec 6, 2022

michalpristas commented Dec 6, 2022

sonarcloud bot commented Dec 6, 2022

michel-laterman commented Dec 6, 2022

jlind23 commented Dec 8, 2022

elasticmachine commented Dec 1, 2022 •

edited

Loading

elasticmachine commented Dec 1, 2022 •

edited

Loading