node-collector schedule prevention on aws fargate

[YevhenVieskov]

Description

I have node-collector pods in pending state. They sceduled on AWS Fargate.
I have added nodeAffinity to kubernetes manifest to prevent pod scheduling on fargate

 ---
 apiVersion: helm.toolkit.fluxcd.io/v2beta1
 kind: HelmRelease
 metadata:
    name: trivy-operator
    namespace: trivy-system
 spec:
    chart:
       spec:
          chart: trivy-operator
          version: 0.11.1
          sourceRef:
             kind: HelmRepository
             name: aqua
             namespace: flux-system
          interval: 1m0s
          values:
             affinity:
                nodeAffinity:
                    requiredDuringSchedulingIgnoredDuringExecution:
                       nodeSelectorTerms:
                       - matchExpressions:
                         - key: eks.amazonaws.com/compute-type
                           operator: NotIn
                           values:
                           - fargate
              targetNamespaces: "mynamespace"
              operator:
                  metricsVulnIdEnabled: true
                  scannerReportTTL: "48h"
                  scanJobsConcurrentLimit: 5
              trivyOperator:
                 scanJobNodeSelector:
                    beta.kubernetes.io/instance-type: m5.large
              trivy:
                offlineScan: true

What did you expect to happen?

I expected node-collector pods would not sceduled to AWS Fargate

What happened instead?

node-collector pods are sceduled to AWS Fargate. They are in pending state.

How to prevent sceduling node-collector pods to AWS Fargate?

[chen-keinan]

@YevhenVieskov A node toleration has been added to to node-collector job , it will be released with v0.38.0

tolerations := []corev1.Toleration{
		{
			Effect:   corev1.TaintEffectNoSchedule,
			Operator: corev1.TolerationOperator(corev1.NodeSelectorOpExists),
		},
		{
			Effect:   corev1.TaintEffectNoExecute,
			Operator: corev1.TolerationOperator(corev1.NodeSelectorOpExists),
		},
		{
			Effect:            corev1.TaintEffectNoExecute,
			Key:               "node.kubernetes.io/not-ready",
			Operator:          corev1.TolerationOperator(corev1.NodeSelectorOpExists),
			TolerationSeconds: pointer.Int64(300),
		},
		{
			Effect:            corev1.TaintEffectNoExecute,
			Key:               "node.kubernetes.io/unreachable",
			Operator:          corev1.TolerationOperator(corev1.NodeSelectorOpExists),
			TolerationSeconds: pointer.Int64(300),
		},

Please check it again with v0.38.0 and update if it solve the problem

[YevhenVieskov]

I updated trivy-operator version.
I have label to fargate node
https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/examples/fargate_profile/main.tf

 fargate_profiles = {
     default = {
         selectors = [
         {
             namespace = "fargate"
             labels = {
                Application = "backend"
             }
         }
         ]
      }
   }

I added label kubernetes.io/hostname: "ip-*" to scanJobNodeSelector .

https://artifacthub.io/packages/helm/trivy-operator/trivy-operator/

trivyOperator:
  scanJobNodeSelector:    
    kubernetes.io/hostname: "ip-*"

I can't create taints for fargate-profile in Terraform, how can I use
node toleration for node-collector job?

[github-actions]

This issue is stale because it has been labeled with inactivity.

[chen-keinan]

@YevhenVieskov how you define Taint on the node ?

[chen-keinan]

Adding support for excluding the nodes by label