OPNET-133: Support remote worker #858

tsorya · 2022-11-15T16:01:00Z

Adding node affinity rule to router deployement that will not allow
router to run on remote worker node (node with custom role label)

openshift-ci · 2022-11-15T16:01:15Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

Miciah

You'll also need to update hashableDeployment and deploymentConfigChanged. Please also add a unit test in TestDeploymentConfigChanged.

Isn't having remote worker nodes already a supported feature? Is the "node-role.kubernetes.io/remote-worker" label new? This change implies that running router pods on nodes with that label is strictly prohibited, but if it was possible to do it before, then it would be a regression to prohibit it now.

tsorya · 2022-11-30T15:08:45Z

It was not possible previously as we don't support remote worker at all for now

Miciah

I have some minor suggestions or questions around comments. Looks fine overall.

pkg/operator/controller/ingress/deployment.go

Miciah · 2022-12-01T21:12:20Z

pkg/operator/controller/ingress/deployment.go

@@ -592,6 +592,21 @@ func desiredRouterDeployment(ci *operatorv1.IngressController, ingressController
 		nodeSelector["node-role.kubernetes.io/master"] = ""
 	default:
 		nodeSelector["node-role.kubernetes.io/worker"] = ""
+		// Disabling running on remote workers


Suggested change

// Disabling running on remote workers

// Disabling running on remote workers.

Is there any intention to allow scheduling routers on remote workers in some future release?

pkg/operator/controller/ingress/deployment_test.go

Miciah · 2022-12-01T21:22:21Z

pkg/operator/controller/names.go

@@ -40,6 +40,8 @@ const (
 	// DefaultCanaryNamespace is the default namespace for
 	// the ingress canary check resources.
 	DefaultCanaryNamespace = "openshift-ingress-canary"
+
+	RemoteWorkerLabel = "node.openshift.io/remote-worker"


Would you mind adding godoc for this definition?

Defining the label here is fine as a temporary measure, but will it eventually be defined in openshift/api or the like?

I hope yes, currently we want to start using it and there is no actual need to add it to openshift-api right a way.
But yes, when we will start increasing usage, we will move it to openshift-api

Adding node affinity rule to router deployement that will not allow router to run on remote worker node (node with custom role label)

Miciah · 2022-12-07T15:34:43Z

Thanks!
/approve
/lgtm

openshift-ci · 2022-12-07T15:39:20Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Miciah

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [Miciah]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

tsorya · 2022-12-07T19:35:20Z

/retest

tsorya · 2022-12-08T07:15:59Z

/retest-required

tsorya · 2022-12-08T09:36:33Z

/retest-required

tsorya · 2022-12-08T11:53:11Z

/retest-required

Miciah · 2022-12-08T21:27:04Z

e2e-aws-operator failed on deprovisioning, which is a known problem with CI on AWS. Because the job failed due to a known issue that is unrelated to this PR and because the e2e-azure-operator and e2e-gcp-operator jobs succeeded, I'm overriding the e2e-aws-operator job.
/override ci/prow/e2e-aws-operator

openshift-ci · 2022-12-08T21:27:24Z

@Miciah: Overrode contexts on behalf of Miciah: ci/prow/e2e-aws-operator

In response to this:

e2e-aws-operator failed on deprovisioning, which is a known problem with CI on AWS. Because the job failed due to a known issue that is unrelated to this PR and because the e2e-azure-operator and e2e-gcp-operator jobs succeeded, I'm overriding the e2e-aws-operator job.
/override ci/prow/e2e-aws-operator

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2022-12-08T21:27:26Z

@tsorya: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

lihongan · 2022-12-14T03:49:37Z

/label qe-approved
follow pre-merge testing and tested with cluster-bot (launch openshift/cluster-ingress-operator#858) and passed, we can see the new nodeAffinity is added to router-default deployment, see

oc -n openshift-ingress get deployment/router-default -oyaml
---
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: node.openshift.io/remote-worker
                operator: NotIn
                values:
                - ""

after adding label node.openshift.io/remote-worker= to one of the worker and scale ingresscontroller/default to 3, then we can see one of router pods cannot be scheduled to that node

$ oc -n openshift-ingress get pod -owide
NAME                              READY   STATUS    RESTARTS   AGE     IP            NODE                                        NOMINATED NODE   READINESS GATES
router-default-574bd8cd79-6vzfm   0/1     Pending   0          4m40s   <none>        <none>                                      <none>           <none>
router-default-574bd8cd79-mgd9b   1/1     Running   0          49m     10.131.0.6    ip-10-0-133-12.us-east-2.compute.internal   <none>           <none>
router-default-574bd8cd79-xv992   1/1     Running   0          6m42s   10.129.2.17   ip-10-0-128-53.us-east-2.compute.internal   <none>           <none>

danmacpherson · 2022-12-15T14:21:27Z

/label docs-approved

CFields651 · 2022-12-19T14:13:18Z

/label px-approved

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 15, 2022

tsorya force-pushed the igal/remote-worker branch from 209252b to 3be2d3f Compare November 15, 2022 20:02

Miciah reviewed Nov 22, 2022

View reviewed changes

tsorya force-pushed the igal/remote-worker branch from 13cf1e9 to a788853 Compare November 30, 2022 15:06

openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 30, 2022

tsorya changed the title ~~WIP: Don't run router on remote worker~~ OPNET-133: Support remote worker Nov 30, 2022

tsorya force-pushed the igal/remote-worker branch from a788853 to b634d2d Compare November 30, 2022 15:08

tsorya marked this pull request as ready for review November 30, 2022 15:08

openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 30, 2022

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 30, 2022

openshift-ci bot requested review from frobware and rfredette November 30, 2022 15:09

tsorya force-pushed the igal/remote-worker branch 2 times, most recently from e614a77 to e1a1e31 Compare December 1, 2022 14:02

Miciah reviewed Dec 1, 2022

View reviewed changes

OPNET-133: Support remote worker

4e7f152

Adding node affinity rule to router deployement that will not allow router to run on remote worker node (node with custom role label)

tsorya force-pushed the igal/remote-worker branch from e1a1e31 to 4e7f152 Compare December 2, 2022 11:39

openshift-ci bot assigned Miciah Dec 7, 2022

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Dec 7, 2022

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 7, 2022

openshift-ci bot added the qe-approved Signifies that QE has signed off on this PR label Dec 14, 2022

openshift-ci bot added the docs-approved Signifies that Docs has signed off on this PR label Dec 15, 2022

openshift-ci bot added the px-approved Signifies that Product Support has signed off on this PR label Dec 19, 2022

openshift-merge-robot merged commit 560b2d3 into openshift:master Dec 19, 2022

lihongan mentioned this pull request Mar 8, 2023

Remote worker openshift/release#36786

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OPNET-133: Support remote worker #858

OPNET-133: Support remote worker #858

tsorya commented Nov 15, 2022 •

edited

Loading

openshift-ci bot commented Nov 15, 2022

Miciah left a comment

tsorya commented Nov 30, 2022

Miciah left a comment

Miciah Dec 1, 2022

Miciah Dec 1, 2022

tsorya Dec 2, 2022

Miciah commented Dec 7, 2022

openshift-ci bot commented Dec 7, 2022

tsorya commented Dec 7, 2022

tsorya commented Dec 8, 2022

tsorya commented Dec 8, 2022

tsorya commented Dec 8, 2022

Miciah commented Dec 8, 2022

openshift-ci bot commented Dec 8, 2022

openshift-ci bot commented Dec 8, 2022

lihongan commented Dec 14, 2022

danmacpherson commented Dec 15, 2022

CFields651 commented Dec 19, 2022

	// Disabling running on remote workers
	// Disabling running on remote workers.

OPNET-133: Support remote worker #858

OPNET-133: Support remote worker #858

Conversation

tsorya commented Nov 15, 2022 • edited Loading

openshift-ci bot commented Nov 15, 2022

Miciah left a comment

Choose a reason for hiding this comment

tsorya commented Nov 30, 2022

Miciah left a comment

Choose a reason for hiding this comment

Miciah Dec 1, 2022

Choose a reason for hiding this comment

Miciah Dec 1, 2022

Choose a reason for hiding this comment

tsorya Dec 2, 2022

Choose a reason for hiding this comment

Miciah commented Dec 7, 2022

openshift-ci bot commented Dec 7, 2022

tsorya commented Dec 7, 2022

tsorya commented Dec 8, 2022

tsorya commented Dec 8, 2022

tsorya commented Dec 8, 2022

Miciah commented Dec 8, 2022

openshift-ci bot commented Dec 8, 2022

openshift-ci bot commented Dec 8, 2022

lihongan commented Dec 14, 2022

danmacpherson commented Dec 15, 2022

CFields651 commented Dec 19, 2022

tsorya commented Nov 15, 2022 •

edited

Loading