Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ScaledObject with Camel K Integration custom resource is broken #4695

Closed
Tracked by #4743
attilamagyar89 opened this issue Jun 16, 2023 · 8 comments · Fixed by #4748
Closed
Tracked by #4743

ScaledObject with Camel K Integration custom resource is broken #4695

attilamagyar89 opened this issue Jun 16, 2023 · 8 comments · Fixed by #4748
Labels
bug Something isn't working

Comments

@attilamagyar89
Copy link

attilamagyar89 commented Jun 16, 2023 •
edited
Loading

Report

We recently updated to Keda 2.10.1, from 2.8.2, and our ScaledObject configuration is now not working correctly. Here is the ScaledObject resource:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: pilot-autoscaler
  namespace: camel-test
  labels:
    camel.apache.org/integration: pilot
    scaledobject.keda.sh/name: pilot-autoscaler
spec:
  maxReplicaCount: 5
  minReplicaCount: 1
  scaleTargetRef:
    apiVersion: camel.apache.org/v1
    kind: Integration
    name: pilot
  triggers:
    - authenticationRef:
        kind: ClusterTriggerAuthentication
        name: keda-trigger-auth-prometheus
      metadata:
        authModes: bearer
        comparisonOperator: GreaterThan
        metricName: application_camel_context_exchanges_inflight_count
        namespace: camel-test
        query: 'application_camel_context_exchanges_inflight_count{job="pilot"}'
        serverAddress: 'https://thanos-querier.openshift-monitoring.svc.cluster.local:9092'
        threshold: '4'
        unsafeSsl: 'false'
      type: prometheus
status:
  conditions:
    - message: Failed to ensure HPA is correctly created for ScaledObject
      reason: ScaledObjectCheckFailed
      status: 'False'
      type: Ready
    - message: ScaledObject check failed
      reason: UnkownState
      status: Unknown
      type: Active
    - status: Unknown
      type: Fallback
  originalReplicaCount: 2
  scaleTargetGVKR:
    group: camel.apache.org
    kind: Integration
    resource: integrations
    version: v1
  scaleTargetKind: camel.apache.org/v1.Integration

And the reffered ClusterTriggerAuthentication:

apiVersion: keda.sh/v1alpha1
kind: ClusterTriggerAuthentication
metadata:
  name: keda-trigger-auth-prometheus
spec:
  secretTargetRef:
    - key: token
      name: thanos-keda-token
      parameter: bearerToken
    - key: ca.crt
      name: thanos-keda-token
      parameter: ca

As you can see, the scaleTargetRef is the Apache Camel K operator's Integration custom resource type, with only one promethes type trigger and ClusterrTriggerAuthentication to the OpenShift monitoring thanos-querier endpoint. The issue is that the HorizontalPodAutoscaler object did not created for this ScaledObject as the status shows. The same ScaledObject configuration with a scaleTargetRef to a standard Deployment still works though.

Expected Behavior

The HPA should be created for the Camel K Integration custom resource with a custom prometheus metrics scaler.

Actual Behavior

The HPA did not created for the Camel K Integration custom resource with a custom prometheus metrics scaler.

Steps to Reproduce the Problem

  1. Install Apache Camel K operator
  2. Create an example Integration
  3. Create a ScaledObject with scaleTargetRef to the example Integration resource, with prometheus type trigger and ClusterTriggerAuthentication.

Logs from KEDA operator

2023-06-16T08:15:07Z	INFO	Creating a new HPA	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"pilot-autoscaler","namespace":"camel-test"}, "namespace": "camel-test", "name": "pilot-autoscaler", "reconcileID": "7410f852-3d16-41e1-a3fc-45b27c606a85", "HPA.Namespace": "camel-test", "HPA.Name": "keda-hpa-pilot-autoscaler"}
2023-06-16T08:15:07Z	ERROR	scale_handler	error resolving auth params	{"type": "ScaledObject", "namespace": "camel-test", "name": "pilot-autoscaler", "scalerIndex": 0, "error": "error parsing prometheus metadata: pod identity cannot be enabled with other auth types"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers
	/remote-source/keda/app/pkg/scaling/scalers_builder.go:81
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).performGetScalersCache
	/remote-source/keda/app/pkg/scaling/scale_handler.go:352
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScalersCache
	/remote-source/keda/app/pkg/scaling/scale_handler.go:273
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).getScaledObjectMetricSpecs
	/remote-source/keda/app/controllers/keda/hpa.go:200
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).newHPAForScaledObject
	/remote-source/keda/app/controllers/keda/hpa.go:74
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).createAndDeployNewHPA
	/remote-source/keda/app/controllers/keda/hpa.go:47
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
	/remote-source/keda/app/controllers/keda/scaledobject_controller.go:408
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
	/remote-source/keda/app/controllers/keda/scaledobject_controller.go:229
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
	/remote-source/keda/app/controllers/keda/scaledobject_controller.go:175
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235
2023-06-16T08:15:07Z	ERROR	Error getting scalers	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"pilot-autoscaler","namespace":"camel-test"}, "namespace": "camel-test", "name": "pilot-autoscaler", "reconcileID": "7410f852-3d16-41e1-a3fc-45b27c606a85", "error": "error parsing prometheus metadata: pod identity cannot be enabled with other auth types"}
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).getScaledObjectMetricSpecs
	/remote-source/keda/app/controllers/keda/hpa.go:202
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).newHPAForScaledObject
	/remote-source/keda/app/controllers/keda/hpa.go:74
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).createAndDeployNewHPA
	/remote-source/keda/app/controllers/keda/hpa.go:47
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
	/remote-source/keda/app/controllers/keda/scaledobject_controller.go:408
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
	/remote-source/keda/app/controllers/keda/scaledobject_controller.go:229
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
	/remote-source/keda/app/controllers/keda/scaledobject_controller.go:175
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235
2023-06-16T08:15:07Z	ERROR	Failed to create new HPA resource	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"pilot-autoscaler","namespace":"camel-test"}, "namespace": "camel-test", "name": "pilot-autoscaler", "reconcileID": "7410f852-3d16-41e1-a3fc-45b27c606a85", "HPA.Namespace": "camel-test", "HPA.Name": "keda-hpa-pilot-autoscaler", "error": "error parsing prometheus metadata: pod identity cannot be enabled with other auth types"}
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).createAndDeployNewHPA
	/remote-source/keda/app/controllers/keda/hpa.go:49
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
	/remote-source/keda/app/controllers/keda/scaledobject_controller.go:408
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
	/remote-source/keda/app/controllers/keda/scaledobject_controller.go:229
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
	/remote-source/keda/app/controllers/keda/scaledobject_controller.go:175
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235
2023-06-16T08:15:07Z	ERROR	Failed to ensure HPA is correctly created for ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"pilot-autoscaler","namespace":"camel-test"}, "namespace": "camel-test", "name": "pilot-autoscaler", "reconcileID": "7410f852-3d16-41e1-a3fc-45b27c606a85", "error": "error parsing prometheus metadata: pod identity cannot be enabled with other auth types"}
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
	/remote-source/keda/app/controllers/keda/scaledobject_controller.go:178
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235
2023-06-16T08:15:07Z	ERROR	Reconciler error	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"pilot-autoscaler","namespace":"camel-test"}, "namespace": "camel-test", "name": "pilot-autoscaler", "reconcileID": "7410f852-3d16-41e1-a3fc-45b27c606a85", "error": "error parsing prometheus metadata: pod identity cannot be enabled with other auth types"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/remote-source/keda/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235

KEDA Version

2.10.1

Kubernetes Version

1.25

Platform

Red Hat OpenShift

Scaler Details

Prometheus

Anything else?

No response
@attilamagyar89 attilamagyar89 added the bug Something isn't working label Jun 16, 2023
@JorTurFer
Copy link
Member

Based on the code, this shouldn't happen but maybe we have missed something

@JorTurFer
Copy link
Member

I can't reproduce the bug. The only way to reproduce the issue has been adding a section in the ClusterTriggerAuthentication the podIdentity section. The ClusterTriggerAuthentication you have shared is exactly how it's in your cluster?

BTW, how are you deploying KEDA? Are you using the open source project or the OpenShift custom metrics server? I don't recognize the folders in the error logs, the errors say:

/remote-source/keda/app/controllers/keda/scaledobject_controller.go:178

and it should be:

/workspace/controllers/keda/scaledobject_controller.go:178

@attilamagyar89
Copy link
Author

We deployed KEDA from the Red Hat operator catalog, exact version is: "Custom Metrics Autoscaler
2.10.1-253 provided by Red Hat"

The full ClusterTriggerAuthentication in our live cluster (with all the metadata fields) is this:

apiVersion: keda.sh/v1alpha1
kind: ClusterTriggerAuthentication
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"keda.sh/v1alpha1","kind":"ClusterTriggerAuthentication","metadata":{"annotations":{},"labels":{"app.kubernetes.io/instance":"openshift-keda"},"name":"keda-trigger-auth-prometheus"},"spec":{"secretTargetRef":[{"key":"token","name":"thanos-keda-token","parameter":"bearerToken"},{"key":"ca.crt","name":"thanos-keda-token","parameter":"ca"}]}}
  resourceVersion: '1256490471'
  name: keda-trigger-auth-prometheus
  uid: 4505fe59-c587-4bf3-86d4-aedd738167cc
  creationTimestamp: '2023-06-12T09:58:36Z'
  generation: 10
  managedFields:
    - apiVersion: keda.sh/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:finalizers':
            .: {}
            'v:"finalizer.keda.sh"': {}
      manager: keda
      operation: Update
      time: '2023-06-12T09:58:36Z'
    - apiVersion: keda.sh/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            .: {}
            'f:kubectl.kubernetes.io/last-applied-configuration': {}
          'f:labels':
            .: {}
            'f:app.kubernetes.io/instance': {}
        'f:spec':
          .: {}
          'f:secretTargetRef': {}
      manager: argocd-controller
      operation: Update
      time: '2023-06-13T15:47:56Z'
  finalizers:
    - finalizer.keda.sh
  labels:
    app.kubernetes.io/instance: openshift-keda
spec:
  secretTargetRef:
    - key: token
      name: thanos-keda-token
      parameter: bearerToken
    - key: ca.crt
      name: thanos-keda-token
      parameter: ca

We already tried to set this in the ClusterTriggerAuthentication:

spec:
  podIdentity: 
    provider: none

But that didn't help.

Our OpenShift cluster runs on-premise, so we do not use any public cloud podintetity provider.

@JorTurFer
Copy link
Member

@zroubalik , FYI.
I can't reproduce the error using 2.10.1 tag, do you know if RH does something else during the build process that could break this?

@zroubalik
Copy link
Member

zroubalik commented Jun 21, 2023 •
edited
Loading

This is a very strange and weird issue. I am not aware of anything in the build process that could cause this, @joelsmith FYI.

I see a configuration option that doesn't make sense for Prometheus: comparisonOperator: GreaterThan what is the reason for this?

Could you please try to use TriggerAuthentication instead of ClusterTriggerAuthentication?

If it doesn't help, could you please try to use upstream KEDA 2.10.1? Just install it via yaml method.

@joelsmith
Copy link
Contributor

I'm tracking this on Red Hat's side as https://issues.redhat.com/browse/OCPBUGS-15274 and I will see if I can figure out what's going on, whether it's specific to Red Hat's build / deploy config or not.

@attilamagyar89
Copy link
Author

"I see a configuration option that doesn't make sense for Prometheus: comparisonOperator: GreaterThan what is the reason for this?"

Fair point, I accidently left the comparisonOperator: GreaterThan in the ScaledObject from a previous attempt. Anyway it does not have any impact on this particular problem as far as I can tell, because after I removed the comparsionOperator, the error in the keda operator pod is still the same.

"Could you please try to use TriggerAuthentication instead of ClusterTriggerAuthentication?"

I tried it, unfortunately it didn't help either.

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: keda-trigger-auth-prometheus-lwi
  namespace: camel-test
spec:
  secretTargetRef:
    - key: token
      name: thanos-keda-token-lwi
      parameter: bearerToken
    - key: ca.crt
      name: thanos-keda-token-lwi
      parameter: ca

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: example-autoscaler-custom
  namespace: camel-test
  labels:
    camel.apache.org/integration: example
    scaledobject.keda.sh/name: example-autoscaler-custom
spec:
  maxReplicaCount: 5
  minReplicaCount: 1
  scaleTargetRef:
    apiVersion: camel.apache.org/v1
    kind: Integration
    name: example
  triggers:
    - authenticationRef:
        kind: TriggerAuthentication
        name: keda-trigger-auth-prometheus-lwi
      metadata:
        authModes: bearer
        metricName: 'application_camel_context_exchanges_inflight_count{job="pilot"}'
        namespace: camel-test
        query: 'application_camel_context_exchanges_inflight_count{job="pilot"}'
        serverAddress: 'https://thanos-querier.openshift-monitoring.svc.cluster.local:9092'
        threshold: '4'
        unsafeSsl: 'false'
      type: prometheus

"If it doesn't help, could you please try to use upstream KEDA 2.10.1? Just install it via yaml method."

Thanks for the suggestion, I managed to uninstall the Red Hat Custom Metrics Autoscaler operator, and install the upstream KEDA 2.10.1 from yaml. Surprisingly (or not surprisingly) the error remained the same, here is the new operator log for my latest attempt:

2023-06-23T12:00:06Z INFO Reconciling ScaledObject {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"example-autoscaler-custom","namespace":"camel-test"}, "namespace": "camel-test", "name": "example-autoscaler-custom", "reconcileID": "cfdebf99-69f8-44ef-8634-be7b0b0bb031"} 2023-06-23T12:00:06Z INFO Creating a new HPA {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"example-autoscaler-custom","namespace":"camel-test"}, "namespace": "camel-test", "name": "example-autoscaler-custom", "reconcileID": "cfdebf99-69f8-44ef-8634-be7b0b0bb031", "HPA.Namespace": "camel-test", "HPA.Name": "keda-hpa-example-autoscaler-custom"} 2023-06-23T12:00:06Z ERROR scale_handler error resolving auth params {"type": "ScaledObject", "namespace": "camel-test", "name": "example-autoscaler-custom", "scalerIndex": 0, "error": "error parsing prometheus metadata: pod identity cannot be enabled with other auth types"} github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers /workspace/pkg/scaling/scalers_builder.go:81 github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).performGetScalersCache /workspace/pkg/scaling/scale_handler.go:352 github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScalersCache /workspace/pkg/scaling/scale_handler.go:273 github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).getScaledObjectMetricSpecs /workspace/controllers/keda/hpa.go:200 github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).newHPAForScaledObject /workspace/controllers/keda/hpa.go:74 github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).createAndDeployNewHPA /workspace/controllers/keda/hpa.go:47 github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists /workspace/controllers/keda/scaledobject_controller.go:408 github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject /workspace/controllers/keda/scaledobject_controller.go:229 github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile /workspace/controllers/keda/scaledobject_controller.go:175 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:122 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:323 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2 /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235 2023-06-23T12:00:06Z ERROR Error getting scalers {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"example-autoscaler-custom","namespace":"camel-test"}, "namespace": "camel-test", "name": "example-autoscaler-custom", "reconcileID": "cfdebf99-69f8-44ef-8634-be7b0b0bb031", "error": "error parsing prometheus metadata: pod identity cannot be enabled with other auth types"} github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).getScaledObjectMetricSpecs /workspace/controllers/keda/hpa.go:202 github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).newHPAForScaledObject /workspace/controllers/keda/hpa.go:74 github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).createAndDeployNewHPA /workspace/controllers/keda/hpa.go:47 github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists /workspace/controllers/keda/scaledobject_controller.go:408 github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject /workspace/controllers/keda/scaledobject_controller.go:229 github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile /workspace/controllers/keda/scaledobject_controller.go:175 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:122 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:323 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2 /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235 2023-06-23T12:00:06Z ERROR Failed to create new HPA resource {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"example-autoscaler-custom","namespace":"camel-test"}, "namespace": "camel-test", "name": "example-autoscaler-custom", "reconcileID": "cfdebf99-69f8-44ef-8634-be7b0b0bb031", "HPA.Namespace": "camel-test", "HPA.Name": "keda-hpa-example-autoscaler-custom", "error": "error parsing prometheus metadata: pod identity cannot be enabled with other auth types"} github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).createAndDeployNewHPA /workspace/controllers/keda/hpa.go:49 github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists /workspace/controllers/keda/scaledobject_controller.go:408 github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject /workspace/controllers/keda/scaledobject_controller.go:229 github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile /workspace/controllers/keda/scaledobject_controller.go:175 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:122 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:323 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2 /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235 2023-06-23T12:00:06Z ERROR Failed to ensure HPA is correctly created for ScaledObject {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"example-autoscaler-custom","namespace":"camel-test"}, "namespace": "camel-test", "name": "example-autoscaler-custom", "reconcileID": "cfdebf99-69f8-44ef-8634-be7b0b0bb031", "error": "error parsing prometheus metadata: pod identity cannot be enabled with other auth types"} github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile /workspace/controllers/keda/scaledobject_controller.go:178 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:122 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:323 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2 /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235 2023-06-23T12:00:06Z ERROR Reconciler error {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"example-autoscaler-custom","namespace":"camel-test"}, "namespace": "camel-test", "name": "example-autoscaler-custom", "reconcileID": "cfdebf99-69f8-44ef-8634-be7b0b0bb031", "error": "error parsing prometheus metadata: pod identity cannot be enabled with other auth types"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:329 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2 /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235

@joelsmith
Copy link
Contributor

joelsmith commented Jun 28, 2023 •
edited
Loading

I think I have figured this out. The new code which gives this error is buggy in that it checks for the empty string instead of "none". I'll post a PR shortly.

Edit: PR posted here: #4748

@joelsmith joelsmith mentioned this issue Jun 28, 2023
Merged
3 tasks
@JorTurFer JorTurFer mentioned this issue Jun 29, 2023
Closed
13 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
Roadmap - KEDA Core
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Bug: Check for "none" when checking pod identity provider joelsmith/keda
4 participants
@joelsmith @zroubalik @JorTurFer @attilamagyar89