Add kep to move kubeproxy out of tree (to staging) #2635

rikatz · 2021-04-18T19:38:14Z

This PR adds the KEP to move kubeproxy to staging, letting people vendor it without vendoring the whole kubernetes/kubernetes repo

rikatz · 2021-04-18T19:38:45Z

/sig network
/assign @thockin @caseydavenport

Casey, Tim, can you be the reviewer/approver of this?

Thanks!!

rikatz · 2021-04-18T19:39:00Z

/cc @aojea @jayunit100

k8s-ci-robot · 2021-04-18T19:50:23Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: rikatz
To complete the pull request process, please ask for approval from caseydavenport and additionally assign ehashman after the PR has been reviewed.
You can assign the PR to them by writing /assign @ehashman in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

keps/prod-readiness/OWNERS
keps/sig-network/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

dims · 2021-04-18T19:54:42Z

keps/sig-network/2634-kubeproxy-staging/README.md

+kube-proxy already have its own repo. The proposal is to:
+
+### Move the internal network dependencies that kube-proxy already has
+kube-proxy have some internal dependencies, mostly notable in `kubernetes/pkg/utils`. We


We need a really really good test suite to make sure these dependencies which we move out can be tested independently of kubernetes. This is the same problem we had with sig-storage's try to move mount utils outside of k/k and eventually we had to bring it back into staging

I think that the proposal is the same as mount utils, create a new staging repo staging/src/k8s.io/network-utils

@aojea line 87 is a section that says UNRESOLVED repository for network utils :)

Thanks @dims

@jayunit100 @aojea you are my reference for testing things, can use some help here! :)

Yes, so here's my point. We need to define where all those things needs to go. My proposal is to move to network-utils, BUT as we need to create a kep for moving kube-proxy to staging, I want to use this to discuss as well where everything should be.

async is the hardest example: @danwinship opened that PR in utils, but by utils guideline async should not be there (because it vendors k8s.io/client-go) so, maybe component-helpers?

We can use this Kep to justify why are we moving things the way we are :)

I'm a huge +1 to move the networking utils to k/net-utils

I'd like to avoid more catch-all "utils" packages if we can avoid it, so we should consider each one independently.

I know we KIND OF want to avoid polluting the "k8s.io" space (e.g. "k8s.io/iptables", "k8s.io/ipvs", "k8s.io/conntrack" all feel somewhat "big". Additionally, we probably don't want to be telling the whole go ecosystem "hey, use this library" because, frankly, we aren't putting that much thought into generalizing theme.

Some alternatives to an omnibus "util" repo.

Make lots of small repos in util.k8s.io/... - we can still keep the source of truth in k/k (the monorepo) and publish to these if we need. "util.k8s.io/iptables" feels less "endorsed".

Make lots of small repos in internal.k8s.io - same as above but even more "warning".

Adopt a convention that makes these libs feel less "big". E.g. "libiptables" or something?

Make lots of small repos in internal.k8s.io - same as above but even more "warning".

That seems plausible...

Adopt a convention that makes these libs feel less "big"

I don't understand your comments about big-ness... Why do we care whether they feel big as opposed to caring whether they are big?

This topic is on the agenda for sig-arch code organization next week (I just missed last week's)

sftim

If a project vendors Kubernetes to import kubeproxy code, this will break them.
On the bright side, afterwards, these importers will have a much cleaner path to
include kubeproxy code. Before moving forward with this plan, we will identify
and communicate to these projects.

SIG Docs' blog subproject can help - and so can the Kubernetes contributor comms team? Would someone like to draft a blog article about this change? Would anyone like help communicating this to the community?

sftim · 2021-05-12T17:11:17Z

keps/sig-network/2634-kubeproxy-staging/README.md

+
+## Proposal
+
+kube-proxy already have its own repo. The proposal is to:


nit:

Suggested change

kube-proxy already have its own repo. The proposal is to:

kube-proxy already has its own repository. The proposal is to:

sftim · 2021-05-12T17:12:23Z

keps/sig-network/2634-kubeproxy-staging/README.md

+"If a project vendors Kubernetes to import kubeproxy code, this will break them. 
+On the bright side, afterwards, these importers will have a much cleaner path to 
+include kubeproxy code. Before moving forward with this plan, we will identify 
+and communicate to these projects." - Shameless copied from 


nit:

Suggested change

"If a project vendors Kubernetes to import kubeproxy code, this will break them.

On the bright side, afterwards, these importers will have a much cleaner path to

include kubeproxy code. Before moving forward with this plan, we will identify

and communicate to these projects." - Shameless copied from

> If a project vendors Kubernetes to import kubeproxy code, this will break them.

> On the bright side, afterwards, these importers will have a much cleaner path to

> include kubeproxy code. Before moving forward with this plan, we will identify

> and communicate to these projects.

–Shamelessly copied from

k8s-triage-robot · 2021-08-10T17:49:23Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

rikatz · 2021-08-10T18:14:48Z

/remove-lifecycle stale
/lifecycle active

k8s-triage-robot · 2021-11-08T19:06:38Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot · 2021-12-08T19:54:54Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

thockin · 2021-12-21T21:50:30Z

keps/sig-network/2634-kubeproxy-staging/README.md

+to future kube-proxy implementations to import this logic without needing to import the 
+whole Kubernetes code.
+
+This was also motivated by [kubernetes #92369](https://github.com/kubernetes/kubernetes/issues/92369)


Should we instead send people to kpng and try to make that the "source of truth" for kube-proxy logic?

Or maybe both? I still rather like the idea that kube-proxy get more encapsulated, but ultimately we don't want multiple implementations of the endpoint selection logic (seeing how complex it has become), so kpng seems like the great leveller.

No opposition to that (even if I had, Jay would kill me!) and also I have the feeling that this enhancement would be much more a buried pipe change than a real need.

Originally this was created to make it easier for the CNI implementations but I guess now every one of the majors one already have their own logics/copy/etc so does this still makes sense?

thockin · 2021-12-21T21:53:00Z

keps/sig-network/2634-kubeproxy-staging/README.md

+kube-proxy already have its own repo. The proposal is to:
+
+### Move the internal network dependencies that kube-proxy already has
+kube-proxy have some internal dependencies, mostly notable in `kubernetes/pkg/utils`. We


I'd like to avoid more catch-all "utils" packages if we can avoid it, so we should consider each one independently.

thockin · 2021-12-21T21:54:27Z

keps/sig-network/2634-kubeproxy-staging/README.md

+vendor the whole kubernetes repo.
+
+<<[UNRESOLVED repository for network utils ]>>
+Although the majority of utils used by kube-proxy can be moved to component-helpers,


let's NOT just keep dumping garbage into component-helpers

thockin · 2021-12-21T22:05:39Z

keps/sig-network/2634-kubeproxy-staging/README.md

+kube-proxy already have its own repo. The proposal is to:
+
+### Move the internal network dependencies that kube-proxy already has
+kube-proxy have some internal dependencies, mostly notable in `kubernetes/pkg/utils`. We


I know we KIND OF want to avoid polluting the "k8s.io" space (e.g. "k8s.io/iptables", "k8s.io/ipvs", "k8s.io/conntrack" all feel somewhat "big". Additionally, we probably don't want to be telling the whole go ecosystem "hey, use this library" because, frankly, we aren't putting that much thought into generalizing theme.

Some alternatives to an omnibus "util" repo.

Make lots of small repos in util.k8s.io/... - we can still keep the source of truth in k/k (the monorepo) and publish to these if we need. "util.k8s.io/iptables" feels less "endorsed".

Make lots of small repos in internal.k8s.io - same as above but even more "warning".

Adopt a convention that makes these libs feel less "big". E.g. "libiptables" or something?

thockin · 2021-12-21T22:07:32Z

keps/sig-network/2634-kubeproxy-staging/README.md

+`kubernetes/kube-proxy/pkg/features` and changing on the code, and putting all the feature
+gates referenced by kube-proxy there
+
+### Move pkg/proxy to staging


Should we also do cmd/kube-proxy?

should we? not sure, I think that right now cmd/kube-proxy is in a good place where it is (following the same approach of kubelet/scheduler/etc) that are there.

But you are here on a really longer time that I am, so I may be missing something really important!!

IMO the split between cmd/kube-proxy and pkg/proxy is pretty much completely artificial. Nothing uses pkg/proxy except cmd/proxy (as far as I caan tell). Repeat for kubelet, controllers, etc and their respective binaries. It just makes the code harder to find.

That said, this is FAR from the most important thing going on right now, so I don't know if it makes sense to do a bunch of busywork?

yeah, so I'm wondering lately, I'm wondering if it's worth to create a whole new kep to split something that probably wont get much attention after kpng.

I still think we need to do some cleanup on pkg/proxy, but probably this can be done without a KEP, and we can do this on a "found this, let's raise a PR and clean" other than discuss a batch/big approach in a KEP.

So should we close this KEP?

thockin · 2022-02-01T07:55:48Z

SGTM

…

On Mon, Jan 31, 2022 at 4:58 PM Ricardo Katz ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In keps/sig-network/2634-kubeproxy-staging/README.md <#2635 (comment)> : > +<<[/UNRESOLVED]>> + +### Move other dependencies kube-proxy already has inside k/k +kube-proxy vendors some packages that are used by other parts of kubernetes codebase. +From now, we can point: +* pkg/util/async (async bounder) which is used by pkg/controlplane +* pkg/util/sysctl which is used by pkg/kubelet and kubemark + +### Move feature gate package dependency +Because kube-proxy relies on defined Feature gates, it vendors `pkg/features`. + +This needs to be copied/moved the same way controller manager already does, moving this to +`kubernetes/kube-proxy/pkg/features` and changing on the code, and putting all the feature +gates referenced by kube-proxy there + +### Move pkg/proxy to staging yeah, so I'm wondering lately, I'm wondering if it's worth to create a whole new kep to split something that probably wont get much attention after kpng. I still think we need to do some cleanup on pkg/proxy, but probably this can be done without a KEP, and we can do this on a "found this, let's raise a PR and clean" other than discuss a batch/big approach in a KEP. So should we close this KEP? — Reply to this email directly, view it on GitHub <#2635 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKWAVCIIFVX4AUFCDCLJ6LUY4V2RANCNFSM43ERO6RA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were assigned.Message ID: ***@***.***>

rikatz · 2022-02-01T10:16:43Z

/close
👋

k8s-ci-robot · 2022-02-01T10:17:05Z

@rikatz: Closed this PR.

In response to this:

/close
👋

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Apr 18, 2021

k8s-ci-robot requested review from caseydavenport and thockin April 18, 2021 19:38

k8s-ci-robot added sig/network Categorizes an issue or PR as relevant to SIG Network. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Apr 18, 2021

k8s-ci-robot assigned caseydavenport and thockin Apr 18, 2021

k8s-ci-robot requested review from aojea and jayunit100 April 18, 2021 19:39

rikatz force-pushed the move-kubeproxy branch from 0f4a142 to 06ba10a Compare April 18, 2021 19:50

dims reviewed Apr 18, 2021

View reviewed changes

Add kep to move kubeproxy to staging

7fa9a59

rikatz force-pushed the move-kubeproxy branch from 06ba10a to 7fa9a59 Compare April 18, 2021 19:55

rikatz changed the title ~~Add kep to move kubeproxy to staging~~ Add kep to move kubeproxy out of tree (to staging) Apr 23, 2021

wojtek-t self-assigned this May 10, 2021

sftim reviewed May 12, 2021

View reviewed changes

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Aug 10, 2021

k8s-ci-robot added lifecycle/active Indicates that an issue or PR is actively being worked on by a contributor. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Aug 10, 2021

k8s-ci-robot added lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. and removed lifecycle/active Indicates that an issue or PR is actively being worked on by a contributor. labels Nov 8, 2021

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Dec 8, 2021

thockin removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Dec 17, 2021

thockin reviewed Dec 21, 2021

View reviewed changes

k8s-ci-robot closed this Feb 1, 2022

danwinship mentioned this pull request Oct 5, 2022

Move kube-proxy networking utilities to staging kubernetes/kubernetes#112886

Closed

astoycos mentioned this pull request Jan 26, 2023

Migrate Shared Kube-Proxy Code into Staging #3786

Closed

danwinship mentioned this pull request Jun 8, 2023

Move pkg/util/iptables to k8s.io/kube-proxy/util/iptables kubernetes/kubernetes#118570

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add kep to move kubeproxy out of tree (to staging) #2635

Add kep to move kubeproxy out of tree (to staging) #2635

rikatz commented Apr 18, 2021

rikatz commented Apr 18, 2021

rikatz commented Apr 18, 2021

k8s-ci-robot commented Apr 18, 2021

dims Apr 18, 2021

aojea Apr 18, 2021

dims Apr 18, 2021

rikatz Apr 18, 2021

rikatz Apr 18, 2021

rikatz Apr 19, 2021

thockin Dec 21, 2021

thockin Dec 21, 2021

danwinship Jan 5, 2022

thockin Jan 10, 2022

sftim left a comment

sftim May 12, 2021

sftim May 12, 2021

k8s-triage-robot commented Aug 10, 2021

rikatz commented Aug 10, 2021

k8s-triage-robot commented Nov 8, 2021

k8s-triage-robot commented Dec 8, 2021

thockin Dec 21, 2021

rikatz Jan 17, 2022

thockin Dec 21, 2021

thockin Dec 21, 2021

thockin Dec 21, 2021

thockin Dec 21, 2021

rikatz Jan 17, 2022

thockin Jan 21, 2022

rikatz Feb 1, 2022

thockin commented Feb 1, 2022 via email

rikatz commented Feb 1, 2022

k8s-ci-robot commented Feb 1, 2022


		## Proposal

		kube-proxy already have its own repo. The proposal is to:

	kube-proxy already have its own repo. The proposal is to:
	kube-proxy already has its own repository. The proposal is to:

-"If a project vendors Kubernetes to import kubeproxy code, this will break them.
-On the bright side, afterwards, these importers will have a much cleaner path to
-include kubeproxy code. Before moving forward with this plan, we will identify
-and communicate to these projects." - Shameless copied from
+> If a project vendors Kubernetes to import kubeproxy code, this will break them.
+> On the bright side, afterwards, these importers will have a much cleaner path to
+> include kubeproxy code. Before moving forward with this plan, we will identify
+> and communicate to these projects.
+&ndash;Shamelessly copied from

Add kep to move kubeproxy out of tree (to staging) #2635

Add kep to move kubeproxy out of tree (to staging) #2635

Conversation

rikatz commented Apr 18, 2021

rikatz commented Apr 18, 2021

rikatz commented Apr 18, 2021

k8s-ci-robot commented Apr 18, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sftim left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

k8s-triage-robot commented Aug 10, 2021

rikatz commented Aug 10, 2021

k8s-triage-robot commented Nov 8, 2021

k8s-triage-robot commented Dec 8, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thockin commented Feb 1, 2022 via email

rikatz commented Feb 1, 2022

k8s-ci-robot commented Feb 1, 2022