-
Notifications
You must be signed in to change notification settings - Fork 39.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add IPv6 support to iptables proxier #48551
Conversation
Hi @leblancd. Thanks for your PR. I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
@k8s-bot ok to test |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Bunch of questions in-line. Mostly minor stuff.
pkg/proxy/iptables/proxier.go
Outdated
if index := strings.LastIndex(s, ":"); index != -1 { | ||
ip := s[0:index] | ||
// Strip off any surrounding brackets. | ||
re := regexp.MustCompile(`\[(.*)\]`) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would a less compute intensive solution be to just check for a '[' at position zero and then take slice w/o first and last character?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, regex seems overpowered for this
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed.
pkg/proxy/iptables/proxier.go
Outdated
// ipPort returns a string of the form "<ip>:<port>" (for IPv4) | ||
// or "[<ip>]:<port>" (for IPv6) for a given IP address and port | ||
func ipPort(ipStr string, port int) string { | ||
ip := net.ParseIP(ipStr) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we know that ipStr will always be an IP address (and not a name)? Is it guaranteed to always be valid?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
net.JoinHostPort() seems to do exactly what this does, if we just fmt.Sprintf("%d", port) or stconv.Itoa(port)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@thockin: Yes, that will do the trick, thanks!
// IPv4 and <ip-address>/128 for IPv6 | ||
func hostAddress(ip net.IP) string { | ||
len := 32 | ||
if ip.To4() == nil { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it always guaranteed that ip is a valid IP address? IOW no need to check for nil? Just wondering if all the callers using this are providing valid IP addresses.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The nil check isn't being done as a sanity check, it's being done to determine if the IP address is IPv4 or IPv6.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would suggest adding additional test cases that exercise the new/modified methods directly (versus through some call chain of higher level function test). Especially as you can exercise error cases.
pkg/proxy/iptables/proxier_test.go
Outdated
{ | ||
name: "V4 UDP glogged error", | ||
epSvcPair: endpointServicePair{ | ||
endpoint: "10.240.0.5:80", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: Should either name all the test case fields or none (consistency). For example, L232 vs L224
pkg/proxy/iptables/proxier_test.go
Outdated
return "IPv6" | ||
} | ||
|
||
func hasJumpTest(t *testing.T, cidrs []string) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can the function name be more descriptive?
pkg/proxy/iptables/proxier_test.go
Outdated
@@ -432,6 +505,36 @@ func hasJump(rules []iptablestest.Rule, destChain, destIP string, destPort int) | |||
} | |||
|
|||
func TestHasJump(t *testing.T) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we make the test name more descriptive or add a docstring to indicate what is being tested?
pkg/proxy/iptables/proxier_test.go
Outdated
@@ -442,18 +545,18 @@ func TestHasJump(t *testing.T) { | |||
"case 1": { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Instead of "case 1", "case 2", ... could these be meaningful names for the test "match 1st rule", "match 2nd rule", "match both dest IP and port",...? Just thinking we can document the failures better.
pkg/proxy/iptables/proxier_test.go
Outdated
proto := strings.ToLower(string(api.ProtocolTCP)) | ||
fwChain := string(serviceFirewallChainName(svcPortName.String(), proto)) | ||
svcChain := string(servicePortChainName(svcPortName.String(), proto)) | ||
//lbChain := string(serviceLBChainName(svcPortName.String(), proto)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull dead code?
pkg/proxy/iptables/proxier_test.go
Outdated
unnamedPort := func(ept *api.Endpoints) { | ||
ept.Subsets = []api.EndpointSubset{{ | ||
Addresses: []api.EndpointAddress{{ | ||
IP: "1.1.1.1", | ||
IP: testIP(1, 1), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I really think the indirection of the testIP function makes the tests harder to read.
/retest |
/assign @dcbw |
pkg/proxy/iptables/proxier.go
Outdated
// Returns just the IP part of an IP:port or IP endpoint string. If the IP | ||
// part is an IPv6 address enclosed in brackets (e.g. "[fd00:1::5]:9999"), | ||
// then the brackets are stripped as well. | ||
func ipPart(s string) string { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Um, how about net.SplitHostPort() instead?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
that seems to DTRT
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, net.SplitHostPort() will help, thanks! But we'll still need a little bit of processing here before it's called. SplitHostPort() only works with strings of the form IPv4-addr:port or [IPv6-addr]:port, so we'll have to first check if the string being processed is of the form IPv4, IPv6-addr, or [IPv6-addr] (and strip off brackets for the latter case) before calling SplitHostPort().
Also the pull-kubernetes-verify test failed; that's likely because of new imports to the testcases and proxy code. A good way to update that is:
And then |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would be nice to see this as individual tiny commits, especially as large as the test changes are.
pkg/proxy/iptables/proxier.go
Outdated
if index := strings.LastIndex(s, ":"); index != -1 { | ||
ip := s[0:index] | ||
// Strip off any surrounding brackets. | ||
re := regexp.MustCompile(`\[(.*)\]`) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, regex seems overpowered for this
pkg/proxy/iptables/proxier.go
Outdated
// Returns just the IP part of an IP:port or IP endpoint string. If the IP | ||
// part is an IPv6 address enclosed in brackets (e.g. "[fd00:1::5]:9999"), | ||
// then the brackets are stripped as well. | ||
func ipPart(s string) string { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
that seems to DTRT
pkg/proxy/iptables/proxier.go
Outdated
// ipPort returns a string of the form "<ip>:<port>" (for IPv4) | ||
// or "[<ip>]:<port>" (for IPv6) for a given IP address and port | ||
func ipPort(ipStr string, port int) string { | ||
ip := net.ParseIP(ipStr) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
net.JoinHostPort() seems to do exactly what this does, if we just fmt.Sprintf("%d", port) or stconv.Itoa(port)
For iptables save and restore operations, kube-proxy currently uses the IPv4 versions of the iptables save and restore utilities (iptables-save and iptables-restore, respectively). For IPv6 operation, the IPv6 versions of these utilities needs to be used (ip6tables-save and ip6tables-restore, respectively). Both this change and PR kubernetes#48551 are needed to get Kubernetes services to work in an IPv6-only Kubernetes cluster (along with setting '--bind-address ::0' on the kube-proxy command line. This change was alluded to in a discussion on services for issue kubernetes#1443. fixes kubernetes#50474
ping me when ready for review. |
We have O(days) to get this into v1.8 |
@thockin Thanks. I think this will have to wait until v1.9. I'm seeing some issues in my functional tests with this code. It might be that the code is okay, but I'm driving it incorrectly. Sorry for the delay on this. |
Adding do-not-merge/release-note-label-needed because the release note process has not been followed. |
I'll post a rebase of the operational (non-unit-test) code with changes for review comments. For UT, I need to figure out how to split up the test cases into several, smaller-sized reviews (and add minimum set of test cases back into this review). |
bdf75b2
to
5f030ad
Compare
5f030ad
to
09936f6
Compare
1d198e1
to
58c6f49
Compare
/assign thockin |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just a couple minor questions, I think.
pkg/proxy/iptables/proxier.go
Outdated
// IP address without brackets | ||
return s | ||
} | ||
if s[0] == '[' && s[len(s)-1] == ']' { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is this a legit notation? Why wouldn't ParseIP
handle it correctly?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You're right, this is not legit. The brackets should only appear for IPv6:port or when IPv6 address appears in a URL. I'll delete these lines.
pkg/proxy/iptables/proxier.go
Outdated
nsn := svcPortName.NamespacedName | ||
if localIPs[nsn] == nil { | ||
localIPs[nsn] = sets.NewString() | ||
if ip := ep.IPPart(); ip != "" { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why would this case happen? If legit, leave a comment?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I suppose this depends on how defensive we want to be with our coding. Here ep.IPPart() could return a null string if net.SplitHostPort() detects bad syntax and returns a null string (along with an error). This can happen (theoretically) if the string in ep.endpoint is not quite right. For example, if the ep.endpoint has too many colons, e.g. "1.2.3.4::9999", or brackets that are out of place, e.g. "1.2.3.4:[0]", then net.SplitHostPort() returns a null string and an error. It's probably very unlikely that we'd end up with a bad format in ep.endpoint at this point, with sanity checks that are done elsewhere. I'll leave this null-string check in for now and add a comment, let me know if you feel this is being too defensive.
58c6f49
to
b6e008a
Compare
The following changes are proposed for the iptables proxier: * There are three places where a string specifying IP:port is parsed using something like this: if index := strings.Index(e.endpoint, ":"); index != -1 { This will fail for IPv6 since V6 addresses contain colons. Also, the V6 address is expected to be surrounded by square brackets (i.e. []:). Fix this by replacing call to Index with call to LastIndex() and stripping out square brackets. * The String() method for the localPort struct should put square brackets around IPv6 addresses. * The logging in the merge() method for proxyServiceMap should put brackets around IPv6 addresses. * There are several places where filterRules destination is hardcoded to /32. This should be a /128 for IPv6 case. * Add IPv6 unit test cases fixes kubernetes#48550
b6e008a
to
5fbc9e4
Compare
For iptables save and restore operations, kube-proxy currently uses the IPv4 versions of the iptables save and restore utilities (iptables-save and iptables-restore, respectively). For IPv6 operation, the IPv6 versions of these utilities needs to be used (ip6tables-save and ip6tables-restore, respectively). Both this change and PR kubernetes#48551 are needed to get Kubernetes services to work in an IPv6-only Kubernetes cluster (along with setting '--bind-address ::0' on the kube-proxy command line. This change was alluded to in a discussion on services for issue kubernetes#1443. fixes kubernetes#50474
This change causes kube-proxy to supply the required "-f ipv6" family flag whenever the conntrack utility is executed and the associated service is using IPv6. This change is required for IPv6-only operation. Note that unit test coverage for the 2-line changes in pkg/proxy/iptables/proxier.go and /pkg/proxy/ipvs/proxier.go will need to be added after support for IPv6 service addresses is added to these files. For pkg/proxy/iptables/proxier.go, this coverage will be added either with PR kubernetes#48551. fixes kubernetes#52027
/lgtm |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: leblancd, thockin Associated issue: 48550 The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these OWNERS Files:
You can indicate your approval by writing |
Automatic merge from submit-queue (batch tested with PRs 50068, 52406, 52394, 48551, 52131). If you want to cherry-pick this change to another branch, please follow the instructions here.. |
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. Add required family flag for conntrack IPv6 operation This change causes kube-proxy to supply the required "-f ipv6" family flag whenever the conntrack utility is executed and the associated service is using IPv6. This change is required for IPv6-only operation. Note that unit test coverage for the 2-line changes in pkg/proxy/iptables/proxier.go and /pkg/proxy/ipvs/proxier.go will need to be added after support for IPv6 service addresses is added to these files. For pkg/proxy/iptables/proxier.go, this coverage will be added either with PR #48551. fixes #52027 **What this PR does / why we need it**: Kube-proxy is currently not supplying the required "-f ipv6" family flag whenever it calls the conntrack utility and the associated service is using an IPv6 service IP address. This means that for IPv6-only operation, conntrack is not properly cleaning up stale UDP connections, and this may be effecting ip6tables operation. **Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes # 52027 **Special notes for your reviewer**: **Release note**: ```release-note NONE ```
Automatic merge from submit-queue (batch tested with PRs 52520, 52033, 53626, 50478). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. Fix kube-proxy to use proper iptables commands for IPv6 operation For iptables save and restore operations, kube-proxy currently uses the IPv4 versions of the iptables save and restore utilities (iptables-save and iptables-restore, respectively). For IPv6 operation, the IPv6 versions of these utilities need to be used (ip6tables-save and ip6tables-restore, respectively). Both this change and PR #48551 are needed to get Kubernetes services to work in an IPv6-only Kubernetes cluster (along with setting '--bind-address ::0' on the kube-proxy command line. This change was alluded to in a discussion on services for issue #1443. fixes #50474 **What this PR does / why we need it**: This change modifies kube-proxy so that it uses the proper commands for iptables save and iptables restore for IPv6 operation. Currently kube-proxy uses 'iptables-save' and 'iptables-restore' regardless of whether it is being used in IPv4 or IPv6 mode. This change fixes kube-proxy so that it uses 'ip6tables-save' and 'ip6tables-restore' commands when kube-proxy is being run in IPv6 mode. **Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #50474 **Special notes for your reviewer**: **Release note**: ```release-note NONE ```
Add IPv6 support to iptables proxier
The following changes are proposed for the iptables proxier:
using something like this:
if index := strings.Index(e.endpoint, ":"); index != -1 {
This will fail for IPv6 since V6 addresses contain colons. Also,
the V6 address is expected to be surrounded by square brackets
(i.e. []:). Fix this by replacing call to Index with
call to LastIndex() and stripping out square brackets.
around IPv6 addresses.
around IPv6 addresses.
/32. This should be a /128 for IPv6 case.
Note: I've left out most of the UT test cases that I had included in my original version of this
PR because the number of lines of code change were much too large for a single review.
I'm including a minimum of UT with this current version of the PR.
fixes #48550
What this PR does / why we need it:
This PR addresses several issues in the iptables proxier for handling IPv6 addresses
that were found via visual code inspection, including:
if index := strings.Index(e.endpoint, ":"); index != -1 {
This will fail for IPv6 since V6 addresses contains many colons, and the V6 address is expected
to be enclosed in square brackets when followed by :.
addresses.
Should be a /128 for IPv6 case.
Which issue this PR fixes (optional, in
fixes #<issue number>(, fixes #<issue_number>, ...)
format, will close that issue when PR gets merged): fixes #48550Special notes for your reviewer:
Release note: