Make nodeport ip configurable #58052

m1093782566 · 2018-01-10T07:51:14Z

What this PR does / why we need it:

By default, kube-proxy accepts everything from NodePort without any filter. It can be a problem for nodes which has both public and private NICs, and people only want to provide a service in private network and avoid exposing any internal service on the public IPs.

This PR makes nodeport ip configurable.

Which issue(s) this PR fixes:
Closes: #21070

Special notes for your reviewer:

Design proposal see: kubernetes/community#1547

Issue in feature repo: kubernetes/enhancements#539

Release note:

Make NodePort IP addresses configurable

m1093782566 · 2018-01-16T04:28:21Z

/assign @thockin @brendandburns

/unassign

k8s-ci-robot · 2018-01-16T06:44:52Z

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
If you have done the above and are still having issues with the CLA being reported as unsigned, please email the CNCF helpdesk: helpdesk@rt.linuxfoundation.org

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

vfreex · 2018-01-17T07:23:20Z

I love this feature.
It would be better if we can alternately specify the whitelisted NIC for incoming traffic if we don't case the IP address and want to share the config file among nodes.

m1093782566 · 2018-01-17T07:27:48Z

@vfreex

I understand, but NIC name is very tricky especially in systemd system, for example, eth0, encap9, ensap7 etc. I doubt NIC name is more portable than CIDR.

brendandburns · 2018-01-17T18:11:02Z

pkg/proxy/apis/kubeproxyconfig/validation/validation.go

+	allErrs := field.ErrorList{}
+LOOP:
+	for i := range nodePortAddresses {
+		switch nodePortAddresses[i] {


Why is this a switch with a single case? shouldn't that be an if?

brendandburns · 2018-01-17T18:13:51Z

pkg/proxy/iptables/proxier.go

+	} else {
+		addressList := addresses.UnsortedList()
+		if len(addressList) == 1 && utilproxy.IsZeroCIDR(addressList[0]) {
+			writeLine(proxier.natRules,


Can we structure this like:

cmd := []string { ... } if len(addressList) && ... { cmd = append(cmd, ...) } else { cmd = append(cmd, ...) } cmd = append(cmd, ...) writeLine(...)

So that we don't have quite so much code duplication?

brendandburns · 2018-01-17T18:15:57Z

pkg/proxy/ipvs/proxier.go

 			if err != nil {
-				glog.Errorf("Failed to get node IP, err: %v", err)
+				glog.Errorf("Failed to get node ip address matching nodeport cidr")


Shouldn't we bail out instead of having the else?

if ... { glog.Errorf(...) return } ...

brendandburns · 2018-01-17T18:17:05Z

This generally looks good to me, but should probably be approved by

@kubernetes/sig-network-api-reviews
@thockin

m1093782566 · 2018-02-26T19:19:10Z

/retest

thockin · 2018-02-26T19:32:29Z

cmd/kube-proxy/app/server.go

@@ -169,6 +169,8 @@ func (o *Options) AddFlags(fs *pflag.FlagSet) {
 		"NAT timeout for TCP connections in the CLOSE_WAIT state")
 	fs.BoolVar(&o.config.EnableProfiling, "profiling", o.config.EnableProfiling, "If true enables profiling via web interface on /debug/pprof handler.")
 	fs.StringVar(&o.config.IPVS.Scheduler, "ipvs-scheduler", o.config.IPVS.Scheduler, "The ipvs scheduler type when proxy mode is ipvs")
+	fs.StringSliceVar(&o.config.NodePortAddresses, "nodeport-addresses", o.config.NodePortAddresses,


This is a FLAG, not code. Someone using the command-line doesn't know what a slice is. You have to tell them how to use it. "A comma-delimited list of IP blocks (e.g. 10.0.0.0/8, 1.2.3.4/32) used to filter addresses local to this node. Defaults to use all local addresses".

ACK, THANKS!

thockin · 2018-02-26T19:33:35Z

pkg/proxy/apis/kubeproxyconfig/validation/validation.go

+			allErrs = append(allErrs, field.Invalid(fldPath, nodePortAddresses, "must be non-empty"))
+			break
+		}
+		if _, _, err := net.ParseCIDR(nodePortAddresses[i]); err != nil {


OK for now. The UX is a little weird, but probably won't get that much use.

thockin

Can be fixed in followup.

thockin · 2018-02-26T19:38:46Z

pkg/proxy/iptables/proxier.go

+		isIPv6 := proxier.iptables.IsIpv6()
+		for address := range addresses {
+			// TODO(thockin, m1093782566): If/when we have dual-stack support we will want to distinguish v4 from v6 zero-CIDRs.
+			if utilproxy.IsZeroCIDR(address) {


need to also test the v6-ness of it. If I specified "10.0.0.0/8,::/0" I would not expect 192.168.1.1 to match.

thockin · 2018-02-26T19:39:26Z

pkg/proxy/iptables/proxier.go

+				break
+			}
+			// Ignore IP addresses with incorrect version
+			if isIPv6 && !conntrack.IsIPv6String(address) || !isIPv6 && conntrack.IsIPv6String(address) {


if isIPv6 != conntrack.IsIPv6String(address)

?

thockin · 2018-02-26T19:41:41Z

pkg/proxy/util/testing/fake.go

+	NetworkInterfaces []net.Interface
+	// The key of map Addrs is the network interface name
+	Address map[string][]net.Addr
+}


add

var _ proxyutil.NetworkInterfacer = &FakeNetwork{}

thockin · 2018-02-26T19:43:57Z

pkg/proxy/util/utils.go

+	}
+	// First round of iteration to pick out `0.0.0.0/0` or `::/0` for the sake of excluding non-zero IPs.
+	for _, cidr := range cidrs {
+		if IsZeroCIDR(cidr) {


This will allow the zero-cidr more than once, Not a big deal, but odd.

thockin · 2018-02-26T20:02:44Z

/lgtm
/approve

k8s-ci-robot · 2018-02-26T20:02:52Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: m1093782566, thockin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~cmd/OWNERS~~ [thockin]
~~pkg/kubemark/OWNERS~~ [thockin]
~~pkg/proxy/OWNERS~~ [thockin]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

fejta-bot · 2018-02-26T22:07:50Z

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel comment for consistent failures.

m1093782566 · 2018-02-27T01:16:17Z

@ixdy

I0226 21:34:57.293] Verifying verify-bazel.sh
I0226 21:34:57.309] 
I0226 21:34:57.310] +++ Running case: verify.bazel 
I0226 21:34:57.312] +++ working dir: /go/src/k8s.io/kubernetes
I0226 21:34:57.313] +++ command: bash "hack/make-rules/../../hack/verify-bazel.sh"
W0226 21:35:10.863] # github.com/bazelbuild/bazel-gazelle/internal/merger
W0226 21:35:10.864] ../../internal/merger/fix.go:93:28: too few values in struct initializer
W0226 21:35:10.864] ../../internal/merger/merger.go:806:16: too few values in struct initializer
W0226 21:35:10.864] ../../internal/merger/merger.go:866:19: too few values in struct initializer
W0226 21:35:10.865] ../../internal/merger/merger.go:870:19: too few values in struct initializer
W0226 21:35:11.056] # github.com/bazelbuild/bazel-gazelle/internal/rules
W0226 21:35:11.056] ../../internal/rules/generator.go:183:16: too few values in struct initializer
W0226 21:35:11.057] ../../internal/rules/sort_labels.go:45:16: too few values in struct initializer
W0226 21:35:11.063] !!! [0226 21:35:11] Call tree:
W0226 21:35:11.065] !!! [0226 21:35:11]  1: ./hack/update-bazel.sh:34 kube::util::go_install_from_commit(...)
W0226 21:35:11.070] !!! [0226 21:35:11] Call tree:
W0226 21:35:11.072] !!! [0226 21:35:11]  1: ./hack/update-bazel.sh:34 kube::util::go_install_from_commit(...)
I0226 21:35:12.105] +++ exit code: 1
I0226 21:35:12.110] +++ error: 1
I0226 21:35:12.132] FAILED   verify-bazel.sh	15s

Seems verify-bazel is broken, PTAL.

m1093782566 · 2018-02-27T01:22:33Z

/retest

ixdy · 2018-02-27T01:23:42Z

/retest

kevin-wangzefeng · 2018-02-27T02:11:29Z

adding milestone as it's for 1.10

k8s-github-robot · 2018-02-27T17:38:41Z

Automatic merge from submit-queue (batch tested with PRs 60430, 60115, 58052, 60355, 60116). If you want to cherry-pick this change to another branch, please follow the instructions here.

m1093782566 · 2018-02-28T01:23:28Z

All outstanding comments will be fixed after freeze.

k8s-ci-robot assigned m1093782566 Jan 10, 2018

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jan 10, 2018

m1093782566 changed the title ~~Nodeip config~~ [WIP]Nodeip config Jan 10, 2018

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jan 10, 2018

m1093782566 force-pushed the nodeip-config branch from ffe4c7b to c7b3378 Compare January 16, 2018 04:27

m1093782566 changed the title ~~[WIP]Nodeip config~~ Make nodeport ip configurable Jan 16, 2018

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 16, 2018

k8s-ci-robot assigned brendandburns and thockin and unassigned m1093782566 Jan 16, 2018

k8s-ci-robot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. and removed cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jan 16, 2018

m1093782566 force-pushed the nodeip-config branch from 357ff11 to 05d7ed1 Compare January 16, 2018 06:47

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Jan 16, 2018

m1093782566 force-pushed the nodeip-config branch 3 times, most recently from 257213e to 04cd7fe Compare January 17, 2018 02:35

brendandburns reviewed Jan 17, 2018

View reviewed changes

m1093782566 added 8 commits February 26, 2018 23:48

validate nodeport-addresses

dbcb2c9

create netwowrk interface util

bf56530

iptables part implementation

ddfa04e

ipvs part implementation

c537ff5

userspace part changes

b498a33

auto generated codes

df7df81

update bazel

9bb4807

fix static checks

2edc97b

m1093782566 force-pushed the nodeip-config branch from a7405ed to 2edc97b Compare February 26, 2018 15:49

thockin reviewed Feb 26, 2018

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 26, 2018

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 26, 2018

thockin added the approved-for-milestone label Feb 26, 2018

k8s-ci-robot added status/approved-for-milestone and removed approved-for-milestone labels Feb 26, 2018

kevin-wangzefeng added this to the v1.10 milestone Feb 27, 2018

k8s-github-robot merged commit 42378ea into kubernetes:master Feb 27, 2018

islinwb mentioned this pull request Mar 5, 2018

Add docs on configuring NodePort IP kubernetes/website#7631

Merged

danwinship mentioned this pull request Jan 3, 2019

Delete invalid assignment in proxy's fuzzer. #69919

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make nodeport ip configurable #58052

Make nodeport ip configurable #58052

m1093782566 commented Jan 10, 2018 •

edited

Loading

m1093782566 commented Jan 16, 2018

k8s-ci-robot commented Jan 16, 2018

vfreex commented Jan 17, 2018

m1093782566 commented Jan 17, 2018

brendandburns Jan 17, 2018

m1093782566 Jan 18, 2018

brendandburns Jan 17, 2018 •

edited

Loading

m1093782566 Jan 18, 2018

brendandburns Jan 17, 2018

m1093782566 Jan 18, 2018

brendandburns commented Jan 17, 2018

m1093782566 commented Feb 26, 2018

thockin Feb 26, 2018 •

edited

Loading

m1093782566 Feb 27, 2018

thockin Feb 26, 2018

thockin left a comment

thockin Feb 26, 2018

thockin Feb 26, 2018

thockin Feb 26, 2018

thockin Feb 26, 2018

thockin commented Feb 26, 2018

k8s-ci-robot commented Feb 26, 2018

fejta-bot commented Feb 26, 2018

m1093782566 commented Feb 27, 2018

m1093782566 commented Feb 27, 2018

ixdy commented Feb 27, 2018

kevin-wangzefeng commented Feb 27, 2018

k8s-github-robot commented Feb 27, 2018

m1093782566 commented Feb 28, 2018

Make nodeport ip configurable #58052

Make nodeport ip configurable #58052

Conversation

m1093782566 commented Jan 10, 2018 • edited Loading

m1093782566 commented Jan 16, 2018

k8s-ci-robot commented Jan 16, 2018

vfreex commented Jan 17, 2018

m1093782566 commented Jan 17, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brendandburns Jan 17, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brendandburns commented Jan 17, 2018

m1093782566 commented Feb 26, 2018

thockin Feb 26, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thockin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thockin commented Feb 26, 2018

k8s-ci-robot commented Feb 26, 2018

fejta-bot commented Feb 26, 2018

m1093782566 commented Feb 27, 2018

m1093782566 commented Feb 27, 2018

ixdy commented Feb 27, 2018

kevin-wangzefeng commented Feb 27, 2018

k8s-github-robot commented Feb 27, 2018

m1093782566 commented Feb 28, 2018

m1093782566 commented Jan 10, 2018 •

edited

Loading

brendandburns Jan 17, 2018 •

edited

Loading

thockin Feb 26, 2018 •

edited

Loading