Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove the 'ingress-ready' node selector and blocking master node scheduling from kind manifests #8406

Closed
wants to merge 2 commits into from
Closed

Remove the 'ingress-ready' node selector and blocking master node scheduling from kind manifests #8406

wants to merge 2 commits into from

Conversation

reegnz
Copy link

@reegnz reegnz commented Mar 31, 2022 •
edited
Loading

The node selector doesn't exist on any other deployment manifest type.
It becomes confusing when someone tries to deploy the nginx-ingress on
a kind cluster and then it won't work out of the box.
This change makes it much easier to use ingress-nginx with kind.

If someone still wants to use a node taint, they're free to apply the
nodeSelector to the deployments with kustomize.

What this PR does / why we need it:

It makes it easier to use ingress-nginx with kind clusters. If someone still wants to use
nodeselectors they're free to do so with kustomize, but a regular kind user
should not be expected to label their nodes unless that's their use-case.

Also this change increases consistency across deployment targets, because this node-selector only exists for kind clusters, without good reason.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation only

Which issue/s this PR fixes

How Has This Been Tested?

I have deployed the manifests to kind.

Checklist:

  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I've read the CONTRIBUTION guide
  • I have added tests to cover my changes.
  • All new and existing tests passed.

@reegnz
Remove the 'ingress-ready' node selector for kind manifests
08702a0 
The node selector doesn't exist on any other deployment manifest type.
It becomes confusing when someone tries to deploy the nginx-ingress on
a kind cluster and then it won't work out of the box.
This change makes it much easier to use ingress-nginx with kind.

If someone still wants to use a node taint, they're free to apply the
nodeSelector to the deployments with kustomize.
@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Mar 31, 2022
@k8s-ci-robot
Copy link
Contributor

@reegnz: This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot
Copy link
Contributor

Welcome @reegnz!

It looks like this is your first PR to kubernetes/ingress-nginx 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/ingress-nginx has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

@k8s-ci-robot k8s-ci-robot added needs-kind Indicates a PR lacks a `kind/foo` label and requires one. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Mar 31, 2022
@k8s-ci-robot
Copy link
Contributor

Hi @reegnz. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added needs-priority size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Mar 31, 2022
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: reegnz
To complete the pull request process, please assign strongjz after the PR has been reviewed.
You can assign the PR to them by writing /assign @strongjz in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@reegnz
Copy link
Author

reegnz commented Mar 31, 2022 •
edited
Loading

The functionality was originally introduced in #5408 and did not exist anywhere beforehand. It only exists for kind clusters, so it should make sense to make the config more consistent with other deployment targets.

@reegnz
Allow scheduling on master nodes for kind clusters
02cae3f 
The restriction is overly strict and for most kind users who run
kind in single-node mode it's overkill (and also makes ingress-nginx
not even start up).
@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Mar 31, 2022
@reegnz reegnz changed the title Remove the 'ingress-ready' node selector from kind manifests Remove the 'ingress-ready' node selector and blocking master node scheduling from kind manifests Mar 31, 2022
@reegnz
Copy link
Author

reegnz commented Mar 31, 2022

I also removed the NoSchedule for master nodes, as most people tend to run kind in single-node mode.

@strongjz
Copy link
Member

strongjz commented Apr 1, 2022

@reegnz can you look into fixing this in the generate-deploy-scripts.sh where these files are generated. This fix will only last for one release till we regenerate those static files with that script.

Thanks,
James

@longwuyuan
Copy link
Contributor

@reegnz , I can run kind and then install ingress-nginx controller with helm or manifest. I can also run make dev-env and I see ingress-nginx installed at the end. So, installation does work out of the box. Hence please show commands and outputs with time, so that someone can copy/paste to reproduce this problem of "not working out of the box" . Also it will really help if you explain clearly what you mean when y. o say it does not work out of the box.

Secondly the changes you are trying to make are not requested by anybody else. If it is your own use-case problem, then it is better to custom install or maintain a fork.

@strongjz @rikatz @tao12345666333 I think we should not accept this change. And definitely not change the generate-deploy to make such a opinionated change like scheduling/taints/labels.

@reegnz
Copy link
Author

reegnz commented Apr 1, 2022 •
edited
Loading

@strongjz sure thing, I'll have a look. just did a grep and did the change, not super familiar with the generation yet, but I'll sort it out.

@longwuyuan The use-case I'm looking for is being able to run the an ingress like this (so the most basic, simplest form of using nginx-ingress with kind):

kind create cluster
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.2/deploy/static/provider/kind/deploy.yaml

If I do that today, this is what I'm getting:

❯ kubectl get pods -n ingress-nginx
NAME                                        READY   STATUS      RESTARTS   AGE
ingress-nginx-admission-create-drxxb        0/1     Completed   0          37s
ingress-nginx-admission-patch-96hs4         0/1     Completed   1          37s
ingress-nginx-controller-56d4b5df54-xlmhx   0/1     Pending     0          37s

And it's pending till the end of time because there won't be a node to schedule for.

The change is so that it's easier to bootstrap a kind cluster with nginx-ingress.

Secondly the changes you are trying to make are not requested by anybody else. If it is your own use-case problem, then it is better to custom install or maintain a fork.

@longwuyuan Well, that felt hostile... way to make someone wanting to contribute to improve thins to feel like crap. Lucky for me that didn't last long.
A feature's worth isn't in it's popularity in my opinion.
I'm still gonna try and upstream this, to improve the simplicity of the deployment to kind! :)

@reegnz
Copy link
Author

reegnz commented Apr 1, 2022 •
edited
Loading

@strongjz I looked into hack/generate-deploy-scripts.sh. It's using hack/manifest-templates/provider/kind/values.yaml which I changed. No other change is necessary.
Ran hack/generate-deploy-scripts.sh and it produced the exact same output that's in this commit. I double-checked, because that didn't seem credible, so I performed the change on main to hack/manifest-templates/provider/kind/values.yaml and it generated the same changes I have here in this PR.

So there's your generate change, it's in hack/manifest-templates/provider/kind/values.yaml :)

@longwuyuan
Copy link
Contributor

longwuyuan commented Apr 1, 2022 via email

@reegnz
Copy link
Author

reegnz commented Apr 1, 2022 •
edited
Loading

Yeah, written communication is difficult. :) No harm done in the end.

I think it works for make dev-env because it wraps kind, and uses a custom config:

ingress-nginx/build/dev-env.sh

Lines 70 to 88 in 2ae0fb8

cat <<EOF | kind create cluster --name ${KIND_CLUSTER_NAME} --image "kindest/node:${K8S_VERSION}" --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
kubeadmConfigPatches:
- |
kind: InitConfiguration
nodeRegistration:
kubeletExtraArgs:
node-labels: "ingress-ready=true"
authorization-mode: "AlwaysAllow"
extraPortMappings:
- containerPort: 80
hostPort: 80
protocol: TCP
- containerPort: 443
hostPort: 443
protocol: TCP

Line 80 is where it configures a node-label for the node-selector.

The basic use-case for kind however is to get rolling with kind create cluster, without a custom config. And the current one requires to distinguish between nodes, which complicates end-users configs, especially if their dev use-case is not a complex multi-node problem.

Also I'm not trying to primarily fix nginx-ingress dev-env here, but the end-users use-case.

@longwuyuan
Copy link
Contributor

longwuyuan commented Apr 1, 2022 via email

@reegnz
Copy link
Author

reegnz commented Apr 1, 2022 •
edited
Loading

I'm running that in an ubuntu VM, so I guess it shouldn't be macos specific? It's on an ubuntu vm.

I'm using lima-vm + metallb + nginx-ingress for my setup btw.
I deploy with argocd + kustomize. but same happens with the commands I've shown too.
I'd rather not involve helm as I've tried to avoid it so far. I'm more a kustomize fan.
I can render a helm chart and then use it with kustomize, but I'd rather the upstream project provide a kustomize base layer for me, which nginx-ingress does well.

@longwuyuan
Copy link
Contributor

longwuyuan commented Apr 1, 2022 via email

@longwuyuan
Copy link
Contributor

Also, just now got the thought that the middle way out is to add a content to the deploy documentation, under the kind installation part, and display the kind config file there and add a config file to the repo for installation on kind.

@strongjz
Copy link
Member

strongjz commented Apr 6, 2022

This was fixed in the latest release, v1.1.3.

/close

@k8s-ci-robot
Copy link
Contributor

@strongjz: Closed this PR.

In response to this:

This was fixed in the latest release, v1.1.3.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@longwuyuan
Copy link
Contributor

longwuyuan commented Apr 7, 2022 via email

@reegnz
Copy link
Author

reegnz commented Apr 11, 2022

@strongjz what exactly fixed this in the latest release? I don't see any change to the manifests, or the docs.

@reegnz
Copy link
Author

reegnz commented Apr 11, 2022 •
edited
Loading

Anyway, my workaround is to simply go with the cloud manifest, and use kustomize to do the bare minimum changes. Still easier than having to kustomize the kind manifest to patch out node selectors.
Would have been nice to simplify the manifest for the common use-case of kind (eg. when just running kind create cluster).

@reegnz reegnz deleted the remove_kind_node_selector branch April 11, 2022 09:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@strongjz strongjz Awaiting requested review from strongjz

@tao12345666333 tao12345666333 Awaiting requested review from tao12345666333

Assignees
No one assigned
Labels
cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-kind Indicates a PR lacks a `kind/foo` label and requires one. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. needs-priority needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@reegnz @k8s-ci-robot @strongjz @longwuyuan