[WIP] Adding taint during image pre-pulling to prevent user pod being scheduled #3011

xcompass · 2023-02-03T18:18:07Z

This is to solve the issue in #3010.

This PR is to add two additional init-containers (first one and last one) for image puller. The first init-container is used to add a taint to prevent user pods being scheduled while the single user image is being pulled. The last init-container removes the taint added by the first one so the user pods can be scheduled on the node. Both init-containers are using the same binary called taintmanager with different parameter for adding and removing the taints.

A new service account is also added to grant permission to call K8s API to manage taint on the node. The taintmanager is using in-cluster service account token to access K8s API. The taint can be customized through helm value.

welcome · 2023-02-03T18:18:10Z

Thanks for submitting your first pull request! You are awesome! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please make sure you followed the pull request template, as this will help us review your contribution more quickly.

You can meet the other Jovyans by joining our Discourse forum. There is also a intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

consideRatio · 2023-02-03T20:52:36Z

Thank you for your work on this @xcompass!! ❤️ 🌻

Can you what you wish to accomplish, and how how it is accomplished in the PR description?

Also, the docker image - where is the Dockerfile and its bundled source code defined?

This will be a quite advanced PR to review, so we will need all the help we can get to consider this.

xcompass · 2023-02-07T05:49:56Z

@consideRatio Please checkout the PR description. The docker image is linked in the description as well. Let me know if there is anything else you would like me to add.

xcompass · 2023-02-07T05:53:30Z

Tagging @yuvipanda @donotpush @Lastcysa for visibility and possible feedbacks

xcompass · 2023-02-07T05:54:44Z

I'll work on failed tests next a few days.

yuvipanda · 2023-02-07T06:21:09Z

Ok, so I absolutely love the idea! I think if we can implement this cleanly, it's a major win to cold-start problems that exist. I too can't seem to find the source for the images, etc though - they should probably live in this repo as well, and be built by chartpress.

The two things to test are:

How does the autoscaler react when the taint is present? Does it go on an infinite loop of adding new nodes because the user nodes can't be scheduled on the tainted nodes? I think figuring this out is going to be important.
What is the failure mode? What happens when the code crashes, and leaves the taint in after the pull too? If the situation resolves itself soon (probably by autoscaler claiming the failed node), I think this is fine.
Is there a latency penalty where the user pod is scheduled onto the node before the tainting can happen?

Anyway, thank you so much for working on this!

consideRatio · 2023-02-07T06:21:25Z

Thank you @xcompass, now I understand the strategy and think your PR description helps others understand the strategy.

I'd like review effort to remain at a very high level still. I'll switch back to #3010 to help narrow down exactly what you see as the problem that you want to address with this. I don't think the issue description captures that at the moment.

yuvipanda · 2023-02-07T06:24:24Z

I would say that importing https://github.com/ubc/taintmanager into https://github.com/jupyterhub/zero-to-jupyterhub-k8s/tree/main/images, and shaping it to be as similar to https://github.com/jupyterhub/zero-to-jupyterhub-k8s/tree/main/images/image-awaiter as possible would also help a lot with getting this reviewed. We have limited go expertise in the jupyterhub community, and making this new go thing be as close to the existing go thing we have will help a lot!

yuvipanda · 2023-02-07T06:28:54Z

The trick that I'm happy to see here and I hadn't considered before is that because initContainers execute sequentially, we can use it to:

Taint nodes,
Pull user image
untaint node

Which massively simplifies the process vs a full controller as I was thinking of before.

yuvipanda · 2023-02-07T06:29:24Z

sorry, I didn't see your comments @consideRatio! I'll let @xcompass provide more info now, as I'd assume they have a more up-to-date understanding of the problem right now than I do :)

yuvipanda

Thanks for adding those here! I am going to be off for a few days, but left a couple preliminary comments to simplify this further, keeping in mind our ability to maintain this long term with limited go expertise.

I think by falling back to kubeconfig auth if we aren't in cluster, we can get rid of the tiltfile as well as all the extra YAML bits needed for testing and do quick tests locally. We will need to add some tests that exercise a z2jh config with this tainting though.

Excited to see progress on this!

yuvipanda · 2023-02-09T05:35:05Z

images/taint-manager/Dockerfile

@@ -0,0 +1,25 @@
+# syntax=docker/dockerfile:1


Atleast to start with and to keep the amount of required go knowledge to a minimum, what do you think about doing this build the same way as https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/main/images/image-awaiter/Dockerfile? I think primarily that means using scratch as the final target rather than distroless.

the reason I'm using distroless is to run taintmanager as nonroot user to be more secure. If it is not a concern, I can change it to scratch.

yuvipanda · 2023-02-09T05:36:21Z

images/taint-manager/taintmanager.go

+    }
+
+    // creates the in-cluster config
+    config, err := rest.InClusterConfig()


Can you fallback to kubeconfig here if inclusterconfig is not available? That would allow us to test this locally much more easily, and get rid of all the extra YAML bits and tiltfile required to simply test it.

Good idea. out-cluster kubeconfig and node name parameter are added.

yuvipanda

Thank you! My next question is how do you think we can write a unit / integration test for this here?

yuvipanda · 2023-02-16T22:34:05Z

images/taint-manager/test/clusterrole.yaml

@@ -0,0 +1,8 @@
+apiVersion: rbac.authorization.k8s.io/v1


Given we can test this by running the taintmanager locally, are these test files still needed?

yuvipanda · 2023-02-16T22:34:33Z

jupyterhub/values.yaml

@@ -559,6 +559,9 @@ scheduling:
        operator: Equal
        value: core
        effect: NoSchedule
+      - key: hub.jupyter.org/imagepulling


image-pull-in-progress maybe?

tfmark · 2023-04-18T08:18:19Z

jupyterhub/schema.yaml

+        required: [enabled]
+        description: |
+          Manage the taints during the single user image prepulling. It will add
+          sepcified taint before image pulling and remove it after it's done.


"sepcified" -> "specified"

mnp · 2024-03-20T13:13:26Z

Did this get dropped? We probably would benefit.

xcompass · 2024-03-20T20:35:51Z

I'll bring it up-to-date people are still interested in it.

yuvipanda · 2024-03-20T20:54:03Z

Yeah i still think this would be pretty useful if we can get it to work correctly!

The taint manager is a go binary to add/remove taint for a node.

This is to add support for out-cluster development and testing. Also removed Titlefile and deployment manifest as this can be tested outside the cluster

xcompass mentioned this pull request Feb 3, 2023

User pod launching timeout with large single user image when cluster scaling up #3010

Open

yuvipanda requested changes Feb 9, 2023

View reviewed changes

xcompass force-pushed the main-taintmanager branch from 42340da to 6521cb0 Compare February 10, 2023 18:53

yuvipanda requested changes Feb 16, 2023

View reviewed changes

tfmark reviewed Apr 18, 2023

View reviewed changes

xcompass added 9 commits April 3, 2024 10:36

Add taintmanager

98fe35a

Add taintmanager toleration to core pods

08c344f

Add taintmanager to schema

674f5a7

Remove GODEBUG from taintmanager env var

fcb060a

Add taint manager

3f65fac

The taint manager is a go binary to add/remove taint for a node.

Update taint manager image name

50bafb9

Reformat to make prettier happy

ad399b4

Add cli parameter for kubeconfig and node name

7da1b85

This is to add support for out-cluster development and testing. Also removed Titlefile and deployment manifest as this can be tested outside the cluster

Fix a typo

c32c90d

xcompass force-pushed the main-taintmanager branch from 0f4323c to c32c90d Compare April 3, 2024 17:37

xcompass changed the title ~~Adding taint during image pre-pulling to prevent user pod being scheduled~~ [WIP] Adding taint during image pre-pulling to prevent user pod being scheduled Apr 3, 2024

Change taintmanager image location to make chartpress happy

c98495a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Adding taint during image pre-pulling to prevent user pod being scheduled #3011

[WIP] Adding taint during image pre-pulling to prevent user pod being scheduled #3011

xcompass commented Feb 3, 2023 •

edited

Loading

welcome bot commented Feb 3, 2023

consideRatio commented Feb 3, 2023 •

edited

Loading

xcompass commented Feb 7, 2023

xcompass commented Feb 7, 2023 •

edited

Loading

xcompass commented Feb 7, 2023

yuvipanda commented Feb 7, 2023

consideRatio commented Feb 7, 2023

yuvipanda commented Feb 7, 2023

yuvipanda commented Feb 7, 2023

yuvipanda commented Feb 7, 2023

yuvipanda left a comment

yuvipanda Feb 9, 2023

xcompass Feb 16, 2023

yuvipanda Feb 9, 2023

xcompass Feb 16, 2023

yuvipanda left a comment

yuvipanda Feb 16, 2023

yuvipanda Feb 16, 2023

tfmark Apr 18, 2023

mnp commented Mar 20, 2024

xcompass commented Mar 20, 2024

yuvipanda commented Mar 20, 2024

[WIP] Adding taint during image pre-pulling to prevent user pod being scheduled #3011

Are you sure you want to change the base?

[WIP] Adding taint during image pre-pulling to prevent user pod being scheduled #3011

Conversation

xcompass commented Feb 3, 2023 • edited Loading

welcome bot commented Feb 3, 2023

consideRatio commented Feb 3, 2023 • edited Loading

xcompass commented Feb 7, 2023

xcompass commented Feb 7, 2023 • edited Loading

xcompass commented Feb 7, 2023

yuvipanda commented Feb 7, 2023

consideRatio commented Feb 7, 2023

yuvipanda commented Feb 7, 2023

yuvipanda commented Feb 7, 2023

yuvipanda commented Feb 7, 2023

yuvipanda left a comment

Choose a reason for hiding this comment

yuvipanda Feb 9, 2023

Choose a reason for hiding this comment

xcompass Feb 16, 2023

Choose a reason for hiding this comment

yuvipanda Feb 9, 2023

Choose a reason for hiding this comment

xcompass Feb 16, 2023

Choose a reason for hiding this comment

yuvipanda left a comment

Choose a reason for hiding this comment

yuvipanda Feb 16, 2023

Choose a reason for hiding this comment

yuvipanda Feb 16, 2023

Choose a reason for hiding this comment

tfmark Apr 18, 2023

Choose a reason for hiding this comment

mnp commented Mar 20, 2024

xcompass commented Mar 20, 2024

yuvipanda commented Mar 20, 2024

xcompass commented Feb 3, 2023 •

edited

Loading

consideRatio commented Feb 3, 2023 •

edited

Loading

xcompass commented Feb 7, 2023 •

edited

Loading