Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐛 awsmachine: only register machine to LB when it's running #5040

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

r4f4
Copy link
Contributor

@r4f4 r4f4 commented Jun 28, 2024

What type of PR is this?

/kind bug

What this PR does / why we need it:

The AWS docs [1] state that to register an instance with the Load Balancer target groups, the instance must be running. Currently, CAPA tries to register it while it's still pending, causing the following error:

0314 22:24:18.512176  701419 awsmachine_controller.go:605] "failed to reconcile LB attachment" err=<
    [could not register machine to load balancer: could not register control plane instance "i-039bb22b1e8df7c99" with load balancer: failed to register instance with target group 'mrb-capa-67-d2pfx-int-22623': InvalidTarget: The following targets are not in a running state and cannot be registered: 'i-039bb22b1e8df7c99'
        status code: 400, request id: 17514354-77ac-42b1-a882-489760563bbd, could not register machine to load balancer: could not register control plane instance "i-039bb22b1e8df7c99" with load balancer: failed to register instance with target group 'mrb-capa-67-d2pfx-ext-6443': InvalidTarget: The following targets are not in a running state and cannot be registered: 'i-039bb22b1e8df7c99'
        status code: 400, request id: 84e1849c-abb7-4af9-9220-8791fdc1a3fb]
 >
I0314 22:24:18.512325  701419 recorder.go:104] "events: Failed to register control plane instance \"i-039bb22b1e8df7c99\" with load balancer: failed to register instance with target group 'mrb-capa-67-d2pfx-ext-6443': InvalidTarget: The following targets are not in a running state and cannot be registered: 'i-039bb22b1e8df7c99'\n\tstatus code: 400, request id: 84e1849c-abb7-4af9-9220-8791fdc1a3fb" type="Warning" object={"kind":"AWSMachine","namespace":"openshift-cluster-api-guests","name":"mrb-capa-67-d2pfx-bootstrap","uid":"58af1162-380b-4f3e-93fe-c0e81401070e","apiVersion":"infrastructure.cluster.x-k8s.io/v1beta2","resourceVersion":"562"} reason="FailedAttachControlPlaneELB"

Even though this doesn't stop the install from succeeding, let's wait for the instance state to be "running" and with that avoid unnecessary AWS API calls.

[1] https://docs.aws.amazon.com/elasticloadbalancing/latest/application/target-group-register-targets.html#register-instances

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #5037

Special notes for your reviewer:

Checklist:

  • squashed commits
  • includes documentation
  • includes emojis
  • adds unit tests
  • adds or updates e2e tests

Release note:

Fix registering instances to Load Balancers when instances are still pending.

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/bug Categorizes issue or PR as related to a bug. labels Jun 28, 2024
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign vincepri for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-priority labels Jun 28, 2024
@k8s-ci-robot
Copy link
Contributor

Hi @r4f4. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jun 28, 2024
Copy link
Contributor

@mtulio mtulio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@k8s-ci-robot
Copy link
Contributor

@mtulio: changing LGTM is restricted to collaborators

In response to this:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@r4f4
Copy link
Contributor Author

r4f4 commented Jun 28, 2024

No regressions found in Openshift e2e CI tests: openshift/installer#8678

@richardcase
Copy link
Member

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jul 1, 2024
@r4f4
Copy link
Contributor Author

r4f4 commented Jul 1, 2024

/hold
I'll look into the failing tests.

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 1, 2024
The AWS docs [1] state that to register an instance with the Load
Balancer target groups, the instance must be running. Currently, CAPA
tries to register it while it's still pending, causing the following
error:

```
0314 22:24:18.512176  701419 awsmachine_controller.go:605] "failed to reconcile LB attachment" err=<
    [could not register machine to load balancer: could not register control plane instance "i-039bb22b1e8df7c99" with load balancer: failed to register instance with target group 'mrb-capa-67-d2pfx-int-22623': InvalidTarget: The following targets are not in a running state and cannot be registered: 'i-039bb22b1e8df7c99'
        status code: 400, request id: 17514354-77ac-42b1-a882-489760563bbd, could not register machine to load balancer: could not register control plane instance "i-039bb22b1e8df7c99" with load balancer: failed to register instance with target group 'mrb-capa-67-d2pfx-ext-6443': InvalidTarget: The following targets are not in a running state and cannot be registered: 'i-039bb22b1e8df7c99'
        status code: 400, request id: 84e1849c-abb7-4af9-9220-8791fdc1a3fb]
 >
I0314 22:24:18.512325  701419 recorder.go:104] "events: Failed to register control plane instance \"i-039bb22b1e8df7c99\" with load balancer: failed to register instance with target group 'mrb-capa-67-d2pfx-ext-6443': InvalidTarget: The following targets are not in a running state and cannot be registered: 'i-039bb22b1e8df7c99'\n\tstatus code: 400, request id: 84e1849c-abb7-4af9-9220-8791fdc1a3fb" type="Warning" object={"kind":"AWSMachine","namespace":"openshift-cluster-api-guests","name":"mrb-capa-67-d2pfx-bootstrap","uid":"58af1162-380b-4f3e-93fe-c0e81401070e","apiVersion":"infrastructure.cluster.x-k8s.io/v1beta2","resourceVersion":"562"} reason="FailedAttachControlPlaneELB"
```

Even though this doesn't stop the install from succeeding, let's wait
for the instance state to be "running" and with that avoid unnecessary
AWS API calls.

[1] https://docs.aws.amazon.com/elasticloadbalancing/latest/application/target-group-register-targets.html#register-instances
@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jul 4, 2024
@r4f4
Copy link
Contributor Author

r4f4 commented Jul 4, 2024

With the updated version:

DEBUG I0704 16:33:28.785101     109 awsmachine_controller.go:968] "registering to v2 load balancer"
DEBUG I0704 16:33:29.293878     109 recorder.go:104] "Cannot register control plane instance \"i-03b7a9be210c6c6d4\" with load balancer: instance is not running" logger="events" type="Warning" object={"kind":"AWSMachine","namespace":"openshift-cluster-api-guests","name":"rdossant-installer-07-vctwp-master-2","uid":"f07dd126-1edf-4c9a-9047-064adcad7e91","apiVersion":"infrastructure.cluster.x-k8s.io/v1beta2","resourceVersion":"551"} reason="FailedAttachControlPlaneELB"
DEBUG I0704 16:33:29.670727     109 awsmachine_controller.go:641] "done reconciling instance" instance={"id":"i-03b7a9be210c6c6d4","instanceState":"pending","type":"m6i.xlarge","subnetId":"subnet-0f01b76318ad9cb2d","imageId":"ami-01d43c90109a44820","securityGroupIds":["sg-0a695045b014e15a9","sg-0e01b50f92a13f13c","sg-073e69c91a0db8276"],"iamProfile":"rdossant-installer-07-vctwp-master-profile","addresses":[{"type":"InternalDNS","address":"ip-10-0-83-84.us-east-2.compute.internal"},{"type":"InternalIP","address":"10.0.83.84"}],"privateIp":"10.0.83.84","enaSupport":true,"ebsOptimized":false,"tags":{"MachineName":"openshift-cluster-api-guests/rdossant-installer-07-vctwp-master-2","Name":"rdossant-installer-07-vctwp-master-2","kubernetes.io/cluster/rdossant-installer-07-vctwp":"owned","sigs.k8s.io/cluster-api-provider-aws/cluster/rdossant-installer-07-vctwp":"owned","sigs.k8s.io/cluster-api-provider-aws/role":"control-plane"},"availabilityZone":"us-east-2c","volumeIDs":["vol-0d1ddd57774d66010"],"instanceMetadataOptions":{"httpEndpoint":"enabled","httpPutResponseHopLimit":1,"httpTokens":"optional","instanceMetadataTags":"disabled"},"privateDnsName":{"enableResourceNameDnsAAAARecord":false,"enableResourceNameDnsARecord":false,"hostnameType":"ip-name"}}
DEBUG I0704 16:33:29.670765     109 awsmachine_controller.go:643] "but find the instance is pending, requeue" instance="i-03b7a9be210c6c6d4"
[...]
DEBUG I0704 16:33:51.738230     109 awsmachine_controller.go:569] "EC2 instance state changed" state="running" instance-id="i-03b7a9be210c6c6d4"
[...]
DEBUG I0704 16:33:55.973309     109 awsmachine_controller.go:968] "registering to v2 load balancer"
DEBUG I0704 16:33:55.973454     109 recorder.go:104] "Control plane instance \"i-03b7a9be210c6c6d4\" is registered with load balancer" logger="events" type="Normal" object={"kind":"AWSMachine","namespace":"openshift-cluster-api-guests","name":"rdossant-installer-07-vctwp-master-2","uid":"f07dd126-1edf-4c9a-9047-064adcad7e91","apiVersion":"infrastructure.cluster.x-k8s.io/v1beta2","resourceVersion":"551"} reason="SuccessfulAttachControlPlaneELB"
DEBUG I0704 16:33:59.098666     109 awsmachine_controller.go:641] "done reconciling instance" instance={"id":"i-03b7a9be210c6c6d4","instanceState":"running","type":"m6i.xlarge","subnetId":"subnet-0f01b76318ad9cb2d","imageId":"ami-01d43c90109a44820","securityGroupIds":["sg-0a695045b014e15a9","sg-0e01b50f92a13f13c","sg-073e69c91a0db8276"],"iamProfile":"rdossant-installer-07-vctwp-master-profile","addresses":[{"type":"InternalDNS","address":"ip-10-0-83-84.us-east-2.compute.internal"},{"type":"InternalIP","address":"10.0.83.84"}],"privateIp":"10.0.83.84","enaSupport":true,"ebsOptimized":false,"tags":{"MachineName":"openshift-cluster-api-guests/rdossant-installer-07-vctwp-master-2","Name":"rdossant-installer-07-vctwp-master-2","kubernetes.io/cluster/rdossant-installer-07-vctwp":"owned","sigs.k8s.io/cluster-api-provider-aws/cluster/rdossant-installer-07-vctwp":"owned","sigs.k8s.io/cluster-api-provider-aws/role":"control-plane"},"availabilityZone":"us-east-2c","volumeIDs":["vol-0d1ddd57774d66010"],"instanceMetadataOptions":{"httpEndpoint":"enabled","httpPutResponseHopLimit":1,"httpTokens":"optional","instanceMetadataTags":"disabled"},"privateDnsName":{"enableResourceNameDnsAAAARecord":false,"enableResourceNameDnsARecord":false,"hostnameType":"ip-name"}}

/test pull-cluster-api-provider-aws-e2e-blocking

@r4f4
Copy link
Contributor Author

r4f4 commented Jul 4, 2024

/test pull-cluster-api-provider-aws-e2e

@r4f4
Copy link
Contributor Author

r4f4 commented Jul 4, 2024

/hold cancel

I don't see any regressions on Openshift e2e tests.

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 4, 2024
Copy link
Contributor

@alexander-demicev alexander-demicev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority ok-to-test Indicates a non-member PR verified by an org member that is safe to test. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Don't register instance to LB target groups unless instance is running
5 participants