gce: Prefer MASTER_ADVERTISE_ADDRESS in apiserver setup #63696

grosskur · 2018-05-11T01:48:41Z

MASTER_ADVERTISE_ADDRESS is used to set the --advertise-address flag
for the apiserver. It's useful for running the apiserver behind a load
balancer.

However, if PROJECT_ID, TOKEN_URL, TOKEN_BODY, and NODE_NETWORK are
all set, the GCE VM's external IP address will be fetched and used
instead and MASTER_ADVERTISE_ADDRESS will be ignored.

Change this behavior so that MASTER_ADVERTISE_ADDRESS takes precedence
because it's more specific. We still fall back to using the VM's
external IP address if the other variables are set.

Also: Move the setting of --ssh-user and --ssh-keyfile based on
PROXY_SSH_USER) to a top-level block because this is common to all
codepaths.

GCE: Fix to make the built-in `kubernetes` service properly point to the master's load balancer address in clusters that use multiple master VMs.

k8s-ci-robot · 2018-05-11T01:48:43Z

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
If you have done the above and are still having issues with the CLA being reported as unsigned, please email the CNCF helpdesk: helpdesk@rt.linuxfoundation.org

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

grosskur · 2018-05-11T18:26:39Z

I've now signed the CNCF CLA.

bowei · 2018-05-13T05:13:50Z

/ok-to-test

bowei · 2018-05-13T05:13:54Z

/assign

bowei · 2018-05-13T05:16:56Z

/assign @dnardo

bowei · 2018-05-13T05:20:13Z

cluster/gce/gci/configure-helper.sh

+    params+=" --advertise-address=${MASTER_ADVERTISE_ADDRESS}"
+  elif [[ -n "${PROJECT_ID:-}" && -n "${TOKEN_URL:-}" && -n "${TOKEN_BODY:-}" && -n "${NODE_NETWORK:-}" ]]; then
+    local -r vm_external_ip=$(curl --retry 5 --retry-delay 3 ${CURL_RETRY_CONNREFUSED} --fail --silent -H 'Metadata-Flavor: Google' "http://metadata/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip")
+    params+=" --advertise-address=${vm_external_ip}"


Seems like this was set only if PROXY_SSH_USER is set in the previous code. Was that wrong?

Not sure. I looked at the git history but couldn't figure out why --advertise-address was only set if PROXY_SSH_USER was set. To be safe, I've updated the diff to leave that entire block as-is.

bowei · 2018-05-13T05:20:30Z

cluster/gce/gci/configure-helper.sh

-    fi
-  elif [ -n "${MASTER_ADVERTISE_ADDRESS:-}" ]; then
-    params="${params} --advertise-address=${MASTER_ADVERTISE_ADDRESS}"
+  if [ -n "${MASTER_ADVERTISE_ADDRESS:-}" ]; then


use the double bracket test to be consistent (e.g. "[[" and "]]")

bowei · 2018-05-13T05:21:23Z

cluster/gce/gci/configure-helper.sh

+  if [ -n "${MASTER_ADVERTISE_ADDRESS:-}" ]; then
+    params+=" --advertise-address=${MASTER_ADVERTISE_ADDRESS}"
+  elif [[ -n "${PROJECT_ID:-}" && -n "${TOKEN_URL:-}" && -n "${TOKEN_BODY:-}" && -n "${NODE_NETWORK:-}" ]]; then
+    local -r vm_external_ip=$(curl --retry 5 --retry-delay 3 ${CURL_RETRY_CONNREFUSED} --fail --silent -H 'Metadata-Flavor: Google' "http://metadata/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip")


why not use get-metadata-value?

Whoops. I initially developed this patch on an older branch that didn't have get-metadata-value. Switched to use get-metadata-value now.

dnardo · 2018-05-14T16:35:22Z

/lgtm other than Bowei's comments.

Was kubeclt logs/exec tested with this change (in all cases) ?

grosskur · 2018-05-15T21:46:45Z

Thanks for the review!

@bowei: I've pushed a new diff that addresses your comments.

@dnardo: I've tested both kubectl exec and kubectl logs in all cases now, and verified they still work.

grosskur · 2018-05-15T23:07:18Z

/retest

MASTER_ADVERTISE_ADDRESS is used to set the --advertise-address flag for the apiserver. It's useful for running the apiserver behind a load balancer. However, if PROJECT_ID, TOKEN_URL, TOKEN_BODY, and NODE_NETWORK are all set, the GCE VM's external IP address will be fetched and used instead and MASTER_ADVERTISE_ADDRESS will be ignored. Change this behavior so that MASTER_ADVERTISE_ADDRESS takes precedence because it's more specific. We still fall back to using the VM's external IP address if the other variables are set. Also: Pass --ssh-user and --ssh-keyfile flags if both PROXY_SSH_USER and MASTER_ADVERTISE_ADDRESS is set.

grosskur · 2018-05-16T01:07:18Z

/test pull-kubernetes-integration

bowei · 2018-05-16T05:31:09Z

/approve no-issue

bowei · 2018-05-16T05:31:33Z

/lgtm

k8s-ci-robot · 2018-05-16T05:31:41Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bowei, dnardo, grosskur

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~cluster/gce/OWNERS~~ [bowei]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-github-robot · 2018-05-16T05:34:00Z

/test all [submit-queue is verifying that this PR is safe to merge]

k8s-github-robot · 2018-05-16T06:25:21Z

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here.

…96-upstream-release-1.8 Automatic merge from submit-queue. Automated cherry pick of #63696: gce: Prefer MASTER_ADVERTISE_ADDRESS in apiserver setup Cherry pick of #63696 on release-1.8. #63696: gce: Prefer MASTER_ADVERTISE_ADDRESS in apiserver setup ```release-note NONE ```

…96-upstream-release-1.10 Automatic merge from submit-queue. Automated cherry pick of #63696: gce: Prefer MASTER_ADVERTISE_ADDRESS in apiserver setup Cherry pick of #63696 on release-1.10. #63696: gce: Prefer MASTER_ADVERTISE_ADDRESS in apiserver setup ```release-note NONE ```

…96-upstream-release-1.9 Automatic merge from submit-queue. Automated cherry pick of #63696: gce: Prefer MASTER_ADVERTISE_ADDRESS in apiserver setup Cherry pick of #63696 on release-1.9. #63696: gce: Prefer MASTER_ADVERTISE_ADDRESS in apiserver setup ```release-note NONE ```

k8s-ci-robot added the release-note-none Denotes a PR that doesn't merit a release note. label May 11, 2018

k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels May 11, 2018

k8s-ci-robot requested review from gmarek and jingax10 May 11, 2018 01:48

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels May 11, 2018

k8s-ci-robot removed the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label May 13, 2018

k8s-ci-robot assigned bowei May 13, 2018

k8s-ci-robot assigned dnardo May 13, 2018

bowei reviewed May 13, 2018

View reviewed changes

grosskur force-pushed the gce-advertise-addr branch from 56188ef to 1433d72 Compare May 15, 2018 21:40

grosskur force-pushed the gce-advertise-addr branch from 1433d72 to 3541a93 Compare May 16, 2018 00:01

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 16, 2018

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 16, 2018

k8s-github-robot merged commit e392f5b into kubernetes:master May 16, 2018

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-none Denotes a PR that doesn't merit a release note. labels May 18, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gce: Prefer MASTER_ADVERTISE_ADDRESS in apiserver setup #63696

gce: Prefer MASTER_ADVERTISE_ADDRESS in apiserver setup #63696

grosskur commented May 11, 2018 •

edited

Loading

k8s-ci-robot commented May 11, 2018

grosskur commented May 11, 2018

bowei commented May 13, 2018

bowei commented May 13, 2018

bowei commented May 13, 2018

bowei May 13, 2018 •

edited

Loading

grosskur May 15, 2018

bowei May 13, 2018

grosskur May 15, 2018

bowei May 13, 2018

grosskur May 15, 2018

dnardo commented May 14, 2018

grosskur commented May 15, 2018

grosskur commented May 15, 2018

grosskur commented May 16, 2018

bowei commented May 16, 2018

bowei commented May 16, 2018

k8s-ci-robot commented May 16, 2018

k8s-github-robot commented May 16, 2018

k8s-github-robot commented May 16, 2018

gce: Prefer MASTER_ADVERTISE_ADDRESS in apiserver setup #63696

gce: Prefer MASTER_ADVERTISE_ADDRESS in apiserver setup #63696

Conversation

grosskur commented May 11, 2018 • edited Loading

k8s-ci-robot commented May 11, 2018

grosskur commented May 11, 2018

bowei commented May 13, 2018

bowei commented May 13, 2018

bowei commented May 13, 2018

bowei May 13, 2018 • edited Loading

Choose a reason for hiding this comment

grosskur May 15, 2018

Choose a reason for hiding this comment

bowei May 13, 2018

Choose a reason for hiding this comment

grosskur May 15, 2018

Choose a reason for hiding this comment

bowei May 13, 2018

Choose a reason for hiding this comment

grosskur May 15, 2018

Choose a reason for hiding this comment

dnardo commented May 14, 2018

grosskur commented May 15, 2018

grosskur commented May 15, 2018

grosskur commented May 16, 2018

bowei commented May 16, 2018

bowei commented May 16, 2018

k8s-ci-robot commented May 16, 2018

k8s-github-robot commented May 16, 2018

k8s-github-robot commented May 16, 2018

grosskur commented May 11, 2018 •

edited

Loading

bowei May 13, 2018 •

edited

Loading