Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix Elastic Agent Fleet Server without TLS e2e tests #6214

Merged
merged 3 commits into from
Feb 24, 2023
Merged

Fix Elastic Agent Fleet Server without TLS e2e tests #6214

merged 3 commits into from
Feb 24, 2023

Conversation

naemono
Copy link
Contributor

@naemono naemono commented Dec 7, 2022 •
edited
Loading

Elastic Agent Fleet Server e2e tests without TLS continue to fail with Agents failing to connect to Fleet Server with:

{"file.name":"cmd/enroll_cmd.go","file.line":469},"message":"Starting enrollment to URL: http://test-fleet-agent-notls-fs-7d99-agent-http.e2e-kzdqg-mercury.svc:8220/","ecs.version":"1.6.0"}
Error: fail to enroll: fail to execute request to fleet-server: dial tcp 10.85.209.164:8220: connect: connection refused

It seems as though Elastic Agent Fleet Server is binding to localhost:8220 from logs:

"bind": "localhost:8220",

This would prevent communication from anything outside of the pod's containers themselves, and would certainly stop other pods from communicating with this port via a service.

This change uses the environment variables defined in the Agent Fleet Documentation to instruct Elastic Agent Fleet Server to listen on 0.0.0.0:8220 instead, and should resolve the communication issues in the e2e tests.

E2E tests passed when running locally with this change.

Will run e2e tests suite in this PR and see how it goes.

--edit--

closes #6367

@naemono
When fleet server is running without TLS, also set FLEET_SERVER_HOST …
5497916 
…+ FLEET_SERVER_PORT.

Update tests to include new required env vars.
@naemono naemono added the >test Related to unit/integration/e2e tests label Dec 7, 2022
@naemono
Copy link
Contributor Author

naemono commented Dec 7, 2022

run/e2e-tests match=TestFleetAgentWithoutTLS

@naemono
Copy link
Contributor Author

naemono commented Dec 8, 2022

This was failing because of an issue with 8.6 Snapshot version of Elastic Agent
elastic/elastic-agent#1867.
Closing

@naemono naemono closed this Dec 8, 2022
@antoineco
Copy link

@naemono as of Elastic Agent v8.6.0 (release version) Fleet Server still binds on localhost:8220 instead of 0.0.0.0:8220: elastic/elastic-agent#2197

Is there any additional option that needs to be set for this to work, like it did until v8.5.x?

@naemono
Copy link
Contributor Author

naemono commented Jan 30, 2023

@antoineco we had recent issues with Agent/Fleet e2e tests, and have had them disabled for a bit, and are just re-enabling them with 8.6.1:

see #6364

Unfortunately, I can't answer this question quite yet until we re-enable, and begin testing again.

From what I'm seeing, there are still issues with forcing localhost: elastic/elastic-agent#2198

@thbkrkr thbkrkr reopened this Jan 31, 2023
@thbkrkr thbkrkr closed this Jan 31, 2023
@thbkrkr thbkrkr mentioned this pull request Jan 31, 2023
Closed
@naemono naemono reopened this Feb 14, 2023
@naemono
Copy link
Contributor Author

naemono commented Feb 14, 2023 •
edited
Loading

run/e2e-tests match=TestFleetAgentWithoutTLS

Running tests manually to test this fix

❯ make clean docker-build docker-push e2e-docker-build e2e-docker-push e2e-run TESTS_MATCH=TestFleetAgentWithoutTLS E2E_TEST_ENV_TAGS=agent E2E_STACK_VERSION=8.6.1

@naemono
Re-enable the actual test.
ddd5c76 
Signed-off-by: Michael Montgomery <mmontg1@gmail.com>
@naemono
Copy link
Contributor Author

naemono commented Feb 14, 2023

Success @thbkrkr:

=== RUN   TestFleetAgentWithoutTLS/K8S_should_be_accessible
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/Label_test_pods
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/Elasticsearch_CRDs_should_exist
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/Webhook_endpoint_should_not_be_empty
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/Remove_Elasticsearch_if_it_already_exists
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/K8S_should_be_accessible#01
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/Label_test_pods#01
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/Kibana_CRDs_should_exist
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/Remove_Kibana_if_it_already_exists
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/K8S_should_be_accessible#02
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/Label_test_pods#02
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/Agent_CRDs_should_exist
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/Remove_Agent_if_it_already_exists
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/K8S_should_be_accessible#03
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/Label_test_pods#03
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/Agent_CRDs_should_exist#01
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/Remove_Agent_if_it_already_exists#01
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/Creating_an_Elasticsearch_cluster_should_succeed
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/Elasticsearch_cluster_should_be_created
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/Creating_Kibana_should_succeed
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/Kibana_should_be_created
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/Creating_an_Agent_should_succeed
=== RUN   TestFleetAgentWithoutTLS/Agent_should_be_created
=== RUN   TestFleetAgentWithoutTLS/Creating_an_Agent_should_succeed#01
=== RUN   TestFleetAgentWithoutTLS/Agent_should_be_created#01
=== RUN   TestFleetAgentWithoutTLS/ES_HTTP_certificate_authority_should_be_set_and_deployed
Retries (30m0s timeout): ..
=== RUN   TestFleetAgentWithoutTLS/ES_transport_certificate_authority_should_be_set_and_deployed
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/All_expected_Pods_should_eventually_be_ready
Retries (30m0s timeout): .................................
=== RUN   TestFleetAgentWithoutTLS/ES_version_should_be_the_expected_one
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/ES_services_should_be_created
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/ES_services_should_have_endpoints
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/Secrets_should_eventually_be_created
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/ES_pods_should_eventually_have_a_certificate
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/ES_cluster_health_should_eventually_be_green
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/Elastic_password_should_be_available
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/Elasticsearch_data_volumes_should_be_of_the_specified_type
=== RUN   TestFleetAgentWithoutTLS/Cluster_should_be_annotated_with_its_UUID_once_bootstrapped
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/ES_nodes_topology_should_eventually_be_the_expected_one
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/ES_version_should_be_the_expected_one#01
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/ES_endpoint_should_eventually_be_reachable
Retries (30m0s timeout): ...
=== RUN   TestFleetAgentWithoutTLS/Check_desired_nodes_API_state
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/Verify_TLS_CA_cert_on_transport_layer_is_the_expected_one
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/Kibana_deployment_should_be_created
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/Kibana_Pods_should_eventually_be_ready
Retries (1h0m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/Kibana_services_should_be_created
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/Kibana_services_should_have_endpoints
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/Secrets_should_eventually_be_created#01
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/Kibana_status_should_be_updated
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/Kibana_should_be_able_to_connect_to_Elasticsearch
Retries (30m0s timeout): ..
=== RUN   TestFleetAgentWithoutTLS/Agent_status_should_be_updated
Retries (30m0s timeout): ............................
=== RUN   TestFleetAgentWithoutTLS/Agent_health_should_be_green
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/ES_data_should_pass_validations
Retries (30m0s timeout): ................................
=== RUN   TestFleetAgentWithoutTLS/Agent_status_should_be_updated#01
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/Agent_health_should_be_green#01
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/ES_data_should_pass_validations#01
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/Deleting_Elasticsearch_should_return_no_error
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/Elasticsearch_should_not_be_there_anymore
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/Elasticsearch_pods_should_eventually_be_removed
Retries (30m0s timeout): ...
=== RUN   TestFleetAgentWithoutTLS/PVCs_should_eventually_be_removed
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/Soft-owned_secrets_should_eventually_be_removed
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/Deleting_Kibana_should_return_no_error
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/Kibana_should_not_be_there_anymore
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/Kibana_pods_should_eventually_be_removed
Retries (30m0s timeout): ............
=== RUN   TestFleetAgentWithoutTLS/Soft-owned_secrets_should_eventually_be_removed#01
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/Deleting_the_resources_should_return_no_error
=== RUN   TestFleetAgentWithoutTLS/The_resources_should_not_be_there_anymore
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/Agent_pods_should_be_eventually_be_removed
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/Deleting_the_resources_should_return_no_error#01
=== RUN   TestFleetAgentWithoutTLS/The_resources_should_not_be_there_anymore#01
Retries (30m0s timeout): .
=== RUN   TestFleetAgentWithoutTLS/Agent_pods_should_be_eventually_be_removed#01
Retries (30m0s timeout): .
--- PASS: TestFleetAgentWithoutTLS (341.90s)
    --- PASS: TestFleetAgentWithoutTLS/K8S_should_be_accessible (0.02s)
    --- PASS: TestFleetAgentWithoutTLS/Label_test_pods (0.04s)
    --- PASS: TestFleetAgentWithoutTLS/Elasticsearch_CRDs_should_exist (0.01s)
    --- PASS: TestFleetAgentWithoutTLS/Webhook_endpoint_should_not_be_empty (0.00s)
    --- PASS: TestFleetAgentWithoutTLS/Remove_Elasticsearch_if_it_already_exists (0.01s)
    --- PASS: TestFleetAgentWithoutTLS/K8S_should_be_accessible#01 (0.02s)
    --- PASS: TestFleetAgentWithoutTLS/Label_test_pods#01 (0.04s)
    --- PASS: TestFleetAgentWithoutTLS/Kibana_CRDs_should_exist (0.01s)
    --- PASS: TestFleetAgentWithoutTLS/Remove_Kibana_if_it_already_exists (0.01s)
    --- PASS: TestFleetAgentWithoutTLS/K8S_should_be_accessible#02 (0.02s)
    --- PASS: TestFleetAgentWithoutTLS/Label_test_pods#02 (0.04s)
    --- PASS: TestFleetAgentWithoutTLS/Agent_CRDs_should_exist (0.01s)
    --- PASS: TestFleetAgentWithoutTLS/Remove_Agent_if_it_already_exists (0.02s)
    --- PASS: TestFleetAgentWithoutTLS/K8S_should_be_accessible#03 (0.02s)
    --- PASS: TestFleetAgentWithoutTLS/Label_test_pods#03 (0.03s)
    --- PASS: TestFleetAgentWithoutTLS/Agent_CRDs_should_exist#01 (0.00s)
    --- PASS: TestFleetAgentWithoutTLS/Remove_Agent_if_it_already_exists#01 (0.02s)
    --- PASS: TestFleetAgentWithoutTLS/Creating_an_Elasticsearch_cluster_should_succeed (0.03s)
    --- PASS: TestFleetAgentWithoutTLS/Elasticsearch_cluster_should_be_created (0.01s)
    --- PASS: TestFleetAgentWithoutTLS/Creating_Kibana_should_succeed (0.02s)
    --- PASS: TestFleetAgentWithoutTLS/Kibana_should_be_created (0.01s)
    --- PASS: TestFleetAgentWithoutTLS/Creating_an_Agent_should_succeed (0.03s)
    --- PASS: TestFleetAgentWithoutTLS/Agent_should_be_created (0.01s)
    --- PASS: TestFleetAgentWithoutTLS/Creating_an_Agent_should_succeed#01 (0.02s)
    --- PASS: TestFleetAgentWithoutTLS/Agent_should_be_created#01 (0.00s)
    --- PASS: TestFleetAgentWithoutTLS/ES_HTTP_certificate_authority_should_be_set_and_deployed (3.01s)
    --- PASS: TestFleetAgentWithoutTLS/ES_transport_certificate_authority_should_be_set_and_deployed (0.00s)
    --- PASS: TestFleetAgentWithoutTLS/All_expected_Pods_should_eventually_be_ready (105.24s)
    --- PASS: TestFleetAgentWithoutTLS/ES_version_should_be_the_expected_one (0.03s)
    --- PASS: TestFleetAgentWithoutTLS/ES_services_should_be_created (0.01s)
    --- PASS: TestFleetAgentWithoutTLS/ES_services_should_have_endpoints (0.01s)
    --- PASS: TestFleetAgentWithoutTLS/Secrets_should_eventually_be_created (0.04s)
    --- PASS: TestFleetAgentWithoutTLS/ES_pods_should_eventually_have_a_certificate (0.05s)
    --- PASS: TestFleetAgentWithoutTLS/ES_cluster_health_should_eventually_be_green (0.01s)
    --- PASS: TestFleetAgentWithoutTLS/Elastic_password_should_be_available (0.01s)
    --- PASS: TestFleetAgentWithoutTLS/Elasticsearch_data_volumes_should_be_of_the_specified_type (0.01s)
    --- PASS: TestFleetAgentWithoutTLS/Cluster_should_be_annotated_with_its_UUID_once_bootstrapped (0.01s)
    --- PASS: TestFleetAgentWithoutTLS/ES_nodes_topology_should_eventually_be_the_expected_one (0.48s)
    --- PASS: TestFleetAgentWithoutTLS/ES_version_should_be_the_expected_one#01 (0.04s)
    --- PASS: TestFleetAgentWithoutTLS/ES_endpoint_should_eventually_be_reachable (7.44s)
    --- PASS: TestFleetAgentWithoutTLS/Check_desired_nodes_API_state (0.16s)
    --- PASS: TestFleetAgentWithoutTLS/Verify_TLS_CA_cert_on_transport_layer_is_the_expected_one (0.02s)
    --- PASS: TestFleetAgentWithoutTLS/Kibana_deployment_should_be_created (0.01s)
    --- PASS: TestFleetAgentWithoutTLS/Kibana_Pods_should_eventually_be_ready (0.01s)
    --- PASS: TestFleetAgentWithoutTLS/Kibana_services_should_be_created (0.01s)
    --- PASS: TestFleetAgentWithoutTLS/Kibana_services_should_have_endpoints (0.01s)
    --- PASS: TestFleetAgentWithoutTLS/Secrets_should_eventually_be_created#01 (0.02s)
    --- PASS: TestFleetAgentWithoutTLS/Kibana_status_should_be_updated (0.02s)
    --- PASS: TestFleetAgentWithoutTLS/Kibana_should_be_able_to_connect_to_Elasticsearch (3.33s)
    --- PASS: TestFleetAgentWithoutTLS/Agent_status_should_be_updated (81.51s)
    --- PASS: TestFleetAgentWithoutTLS/Agent_health_should_be_green (0.01s)
    --- PASS: TestFleetAgentWithoutTLS/ES_data_should_pass_validations (98.26s)
    --- PASS: TestFleetAgentWithoutTLS/Agent_status_should_be_updated#01 (0.01s)
    --- PASS: TestFleetAgentWithoutTLS/Agent_health_should_be_green#01 (0.00s)
    --- PASS: TestFleetAgentWithoutTLS/ES_data_should_pass_validations#01 (0.00s)
    --- PASS: TestFleetAgentWithoutTLS/Deleting_Elasticsearch_should_return_no_error (0.01s)
    --- PASS: TestFleetAgentWithoutTLS/Elasticsearch_should_not_be_there_anymore (0.00s)
    --- PASS: TestFleetAgentWithoutTLS/Elasticsearch_pods_should_eventually_be_removed (6.03s)
    --- PASS: TestFleetAgentWithoutTLS/PVCs_should_eventually_be_removed (0.01s)
    --- PASS: TestFleetAgentWithoutTLS/Soft-owned_secrets_should_eventually_be_removed (0.01s)
    --- PASS: TestFleetAgentWithoutTLS/Deleting_Kibana_should_return_no_error (0.01s)
    --- PASS: TestFleetAgentWithoutTLS/Kibana_should_not_be_there_anymore (0.00s)
    --- PASS: TestFleetAgentWithoutTLS/Kibana_pods_should_eventually_be_removed (33.08s)
    --- PASS: TestFleetAgentWithoutTLS/Soft-owned_secrets_should_eventually_be_removed#01 (0.00s)
    --- PASS: TestFleetAgentWithoutTLS/Deleting_the_resources_should_return_no_error (0.02s)
    --- PASS: TestFleetAgentWithoutTLS/The_resources_should_not_be_there_anymore (0.02s)
    --- PASS: TestFleetAgentWithoutTLS/Agent_pods_should_be_eventually_be_removed (0.01s)
    --- PASS: TestFleetAgentWithoutTLS/Deleting_the_resources_should_return_no_error#01 (0.05s)
    --- PASS: TestFleetAgentWithoutTLS/The_resources_should_not_be_there_anymore#01 (0.04s)
    --- PASS: TestFleetAgentWithoutTLS/Agent_pods_should_be_eventually_be_removed#01 (0.01s)
PASS
ok  	github.com/elastic/cloud-on-k8s/v2/test/e2e/agent	341.948s

thbkrkr
thbkrkr approved these changes Feb 22, 2023
View reviewed changes
Copy link
Contributor

@thbkrkr thbkrkr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Do we need to document that Fleet >= 8.6.1 without TLS works only from ECK 2.7.0?

@thbkrkr thbkrkr added the v2.7.0 label Feb 22, 2023
@naemono
Copy link
Contributor Author

naemono commented Feb 24, 2023

Do we need to document that Fleet >= 8.6.1 without TLS works only from ECK 2.7.0?

Decided to do nothing, as we still have fleet/agent e2e tests disabled due to #6331

@naemono naemono merged commit 9611f9b into elastic:main Feb 24, 2023
@naemono naemono deleted the fleet-agent-add-fleet-server-host-ip branch February 24, 2023 15:29
@thbkrkr thbkrkr mentioned this pull request Mar 7, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@thbkrkr thbkrkr thbkrkr approved these changes

Assignees
No one assigned
Labels
>test Related to unit/integration/e2e tests v2.7.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

TestFleetAgentWithoutTLS is failing
3 participants
@naemono @antoineco @thbkrkr