Support inter-Pod affinity to one or more Pods

[bsalamat]

In the current implementation of inter-Pod affinity, the scheduler looks for a single existing pod that can satisfy all the terms of inter-pod affinity of an incoming pod.
With the recent changes made to the implementation of inter-Pod affinity, we can now support multiple pods satisfying inter-pod affinity. One of the main reasons we didn't pursue the idea before was the fact that the inter-pod affinity feature was very slow (3 orders of magnitude slower than other scheduler predicates). We didn't want to add more complication to an already slow predicate. However, we can now think about adding the feature.
With this feature, a pod can have multiple affinity terms satisfied by a group of pods, as opposed to only a single pod. For example:

- Assume that the cluster has two nodes: 
- nodeA located in in zone1/region1
- nodeB located in zone2/region1

- There are two existing pods on these nodes:
- Pod1:
    - nodeName: "nodeA"
    - label "foo":""

- Pod2:
    - nodeName: "nodeB"
    - label "bar":""

- Pod3 comes in with inter-pod affinity:
    - affinity terms:
        - {label "foo" exists, topologyKey: "region"}
        - {label "bar" exists, topologyKey: "zone"}

With our current (K8s 1.12) implementation, Pod3 is not schedulable, because there is no single pod that satisfies all of its affinity terms. However, if we support multiple pods satisfying the affinity terms, Pod3 can be scheduled on nodeB. Pod1 satisfies the first term of its affinity in region1 and Pod2 satisfies its second term in zone2. So, any node in zone2/region1 will be feasible for Pod3.

Given the current implementation of inter-pod affinity and the use of "Topology Pair Maps", I believe implementing this feature requires little changes and won't have noticeable performance impact.

/kind feature
/sig scheduling

cc/ @Huang-Wei @ahmad-diaa

[bsalamat]

It is worth noting that this will apply only to inter-pod affinity, not inter-pod anti-affinity. Inter-pod anti-affinity is considered "violated" if there is a pod that matches ANY terms of the anti-affinity. So, matching against a group of Pods does not make sense for anti-affinity.

[Huang-Wei]

@bsalamat I have a full picture on affinity/anti-affinity code after delivering #68173. I'm more than happy to help on this :)

/assign

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[misterikkit]

This question was probably answered elsewhere, but could the change in behavior disrupt existing clusters?

e.g. a canary workload is launched with pod-affinity for labels {app="foo", env="canary"}. That workload could end up in a topology containing {app="foo", env="prod"} & {app="bar", env="canary"} after this change.

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[Huang-Wei]

/remove-lifecycle rotten

[Huang-Wei]

Re @misterikkit:

a canary workload is launched with pod-affinity for labels {app="foo", env="canary"}. That workload could end up in a topology containing {app="foo", env="prod"} & {app="bar", env="canary"} after this change.

If the goal is to gather pods with labels {app="foo", env="canary"}, the workload should have it defined within the same affinityTerm, in different expressions.

And if app="foo" and env="canary" are defined in two different affinityTerms, then yes, after the change, a topology containing {app="foo", env="prod"} & {app="bar", env="canary"} can be a fit.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[Huang-Wei]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[Huang-Wei]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[Huang-Wei]

I'd suggest putting it on our plate only when there is a couple of viable practical use cases.

[sanposhiho]

Quoting from the request:

Suppose we got an incoming pod with 2 pod affinity requirements. One is region and the other is zone. E.g the first one is for affinity to some light-weight RPC service it depends on, while the second for affinity to some heavy-weight storage access.

It makes sense. To be more general, when people want to make sure dependent services exist in the same domain and the pod has more than one kind of dependent services (like the above example), then they cannot use today’s PodAffinity to achieve that.

[alculquicondor]

I know it makes sense, but there doesn't seem to be enough pressure to get this done. This issue has been opened since 2018, and there aren't more people asking for it. Maybe you can collect some feedback sending an email to the mailing list?

[sanposhiho]

👍 In sig-scheduling mailing list? or is there another suitable one?

[Huang-Wei]

yes, and discuss.k8s.io

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sanposhiho]

/remove-lifecycle rotten
/reopen

[k8s-ci-robot]

@sanposhiho: Reopened this issue.

In response to this:

/remove-lifecycle rotten
/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[fentas]

Curious shouldn't this be possible by reading the design proposal?
I assumed (interpreted it) that each entry in requiredDuringSchedulingIgnoredDuringExecution looks up a set of pods and these pods reduce the set of nodes. Kind of confused why this is even an array then, as either matchExpressions or matchLabels can match multiple things. Why would I define a new entry in requiredD.. when I can add an expression in matchExpressions itself to match the same pod?
Also, it is nowhere written that you can't target multiple pods (or I over-/missread).

But it seems not, so my use case is related to:

• Cilium unable to remove taints on Azure AKS cilium/cilium#19788
• AKS issue for taint removal Azure/AKS#2934

We're running cilium in chaining with aks CSI. But since mid-last year removal of taints is only possible via Azure API breaking the cilium node readiness.
Right now, we're unable to run AKS in BYOCNI mode, unless it leaves its preview phase with Cilium.

Following this comment from Microsoft we followed the advice and created a mutation webhook (instead of calling the API) adding a podAffinity so that pods only get scheduled on nodes where cilium agent is already running.
But this breaks now other podAffinities.

[alculquicondor]

I can't speak for the original intent of the design, as it predates my time here, but the reality is that it's not implemented like that. Then we can't change the behavior as it would be backwards-incompatible.

So two things:

• Docs need to be updated to reflect the current implementation. Feel free to open one issue or I'll do so when I have a chance.
• Is there still justification for this feature? In your case, the cilium agent is probably a daemonset, so it already is designed to run in a set of nodes that have a label. Then the pods can use node affinity, instead of pod affinity. Also, node affinity is faster to calculate, in case that matters to you.

[fentas]

Thanks for the feedback.
node affinity won't work in this use case.

The idea is/was to make sure that cilium agent (daemonset) is scheduled first before any other pod is scheduled.
Normally this happens via taints and the agent removes the taint but Microsoft broke or rather disabled this functionality.

Our mutation webhook just merged this podAffinity to every pod created, but this breaks now any other podAffinity already set on the pod itself.

# resulting in this
  affinity:
    podAffinity:
      requiredduringschedulingignoredduringexecution:
      - labelSelector:
          matchLabels:
            example.com/name: myservice
        topologyKey: kubernetes.io/hostname
      - labelSelector:
          matchExpressions:
          - key: k8s-app
            operator: In
            values:
            - cilium
        namespaces:
        - kube-system
        topologyKey: kubernetes.io/hostname

[kerthcet]

Sorry, a bit confusion here, doesn't this works as expected? Unless the cilium pod launches, this new created pod will stuck in pending?

[alculquicondor]

@kerthcet the problem is that kube-scheduler looks for a single pod that satisfies all affinities.

Well, even if we add the feature that you request, it would only be available in 1.28 at the earliest (potentially 1.29, as the feature has to be disabled by default first). So I would suggest you open an issue against Azure support.

Other than that, you are welcome to work on this feature.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.