Flux v2 reconciler is not working correctly for HTTPScaledObject

[mvanhil]

Report

Flux v2 on an AKS 1.29.4.
Flux reports the following error:

"message": "Health check failed after 1m0.025810671s: timeout waiting for: [HTTPScaledObject/daprpoc/http-scaledobject-orderapplication status: 'InProgress']",

Current YAML manifest:

kind: HTTPScaledObject
apiVersion: http.keda.sh/v1alpha1
metadata:
  name: http-scaledobject-orderapplication
  namespace: daprpoc
spec:
  hosts: 
    - dev-app-ordersample.example.com
  pathPrefixes:
    - /
  scaleTargetRef:
    name: orderapplication
    kind: Deployment
    apiVersion: apps/v1
    service: orderapplication-svc
    port: 80
  replicas:
    min: 0     # We want to scale to zero eventually
    max: 10
  scaledownPeriod: 300
  scalingMetric: # requestRate and concurrency are mutually exclusive
    requestRate:
      granularity: 1s
      targetValue: 100
      window: 1m
    concurrency:
      targetValue: 100

kubectl -n daprpoc describe httpscaledobject http-scaledobject-orderapplication

reports the following:

Status:
  Conditions:
    Message:        Identified HTTPScaledObject creation signal
    Reason:         PendingCreation
    Status:         Unknown
    Timestamp:      2024-05-29T14:34:19Z
    Type:           Ready
    Message:        App ScaledObject created
    Reason:         AppScaledObjectCreated
    Status:         True
    Timestamp:      2024-05-29T14:34:19Z
    Type:           Ready
    Message:        Finished object creation
    Reason:         HTTPScaledObjectIsReady
    Status:         True
    Timestamp:      2024-05-29T14:34:19Z
    Type:           Ready
  Target Service:   orderapplication-svc:80
  Target Workload:  apps/v1/Deployment/orderapplication

Expected Behavior

Reconciliation to work correctly

Actual Behavior

Health check on HTTPScaledObject failed

Steps to Reproduce the Problem

1. Use fluxv2 to deploy your manifest files

Logs from KEDA HTTP operator

2024-05-30T08:20:11Z    INFO    Reconciling ScaledObject        {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"azure-service-bus-scaledobject","namespace":"daprpoc"}, "namespace": "daprpoc", "name": "azure-service-bus-scaledobject", "reconcileID": "cdaf8f3d-0334-4fe2-aa2c-6bbe3042379a"}
2024-05-30T08:20:12Z    INFO    Reconciling ScaledObject        {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"http-scaledobject-orderapplication","namespace":"daprpoc"}, "namespace": "daprpoc", "name": "http-scaledobject-orderapplication", "reconcileID": "6568fd95-0002-4e37-94f9-846e9813b865"}
2024-05-30T08:20:27Z    INFO    Reconciling ScaledObject        {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"azure-service-bus-scaledobject","namespace":"daprpoc"}, "namespace": "daprpoc", "name": "azure-service-bus-scaledobject", "reconcileID": "a4ee9e07-94d3-4283-af57-fa3d18f6273a"}
2024-05-30T08:20:45Z    INFO    Reconciling ScaledObject        {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"keda-add-ons-http-interceptor","namespace":"kube-system"}, "namespace": "kube-system", "name": "keda-add-ons-http-interceptor", "reconcileID": "3b1a6504-ca4f-4e8c-9831-d58930deb19a"}
2024-05-30T08:21:13Z    INFO    Reconciling ScaledObject        {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"http-scaledobject-orderapplication","namespace":"daprpoc"}, "namespace": "daprpoc", "name": "http-scaledobject-orderapplication", "reconcileID": "9f7925ce-6727-44da-8fbd-293816a1f431"}

PS: it's only the http-scaledobject-orderapplication that does not report correctly

HTTP Add-on Version

0.8.0

Kubernetes Version

1.29

Platform

Microsoft Azure

Anything else?

[JorTurFer]

Hello,

You have to choose between requestRate and concurrency. They are mutually exclusive and probably that's the reason because the resource is pending

  scalingMetric: # requestRate and concurrency are mutually exclusive
    requestRate:
      granularity: 1s
      targetValue: 100
      window: 1m
    concurrency:
      targetValue: 100

[mvanhil]

Hi @JorTurFer
Thanks for that, but it didn't resolve the issue unfortunately.

Still getting the message:
{ "lastTransitionTime": "2024-06-07T08:22:23+00:00", "message": "Health check failed after 5m0.032530515s: timeout waiting for: [HTTPScaledObject/daprpoc/http-scaledobject-orderapplication status: 'InProgress']", "reason": "HealthCheckFailed", "status": "False", "type": "Ready" },

kind: HTTPScaledObject
apiVersion: http.keda.sh/v1alpha1
metadata:
  name: http-scaledobject-orderapplication
  namespace: daprpoc
spec:
  hosts: 
    - dev-app-ordersample.example.be
  pathPrefixes:
    - /
  scaleTargetRef:
    name: orderapplication
    kind: Deployment
    apiVersion: apps/v1
    service: orderapplication-svc
    port: 80
  replicas:
    min: 0     # We want to scale to zero eventually
    max: 10
  scaledownPeriod: 300
  scalingMetric: # requestRate and concurrency are mutually exclusive
    # requestRate:
    #   granularity: 1s
    #   targetValue: 100
    #   window: 1m
    concurrency:
      targetValue: 100

[mvanhil]

Anything related? #611

[JorTurFer]

Sorry for the slow response, I've been involved on a huge working peak :(
Do you see any error in in the HTTP Add-on logs? (there are 2 operators, we need to http add-on one in this case)

[mvanhil]

No worries, already happy you want to help :-)

Which logs would you like to see?

I don't see any errors in any of these.

However; if I trigger a scale up by browsing to the application (so replica 0 -> 1):

2024-06-24T13:51:51Z    INFO    scaleexecutor   Successfully updated ScaleTarget        {"scaledobject.Name": "http-scaledobject-orderapplication", "scaledObject.Namespace": "daprpoc", "scaleTarget.Name": "orderapplication", "Original Replicas Count": 0, "New Replicas Count": 1}
2024-06-24T13:51:59Z    INFO    Reconciling ScaledObject        {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"http-scaledobject-orderapplication","namespace":"daprpoc"}, "namespace": "daprpoc", "name": "http-scaledobject-orderapplication", "reconcileID": "3c49b97f-a92b-40d3-8ae2-0db81843b006"}
2024-06-24T13:52:00Z    INFO    Reconciling ScaledObject        {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"keda-add-ons-http-interceptor","namespace":"kube-system"}, "namespace": "kube-system", "name": "keda-add-ons-http-interceptor", "reconcileID": "a1f5f811-29e0-4e13-af4c-fb02748b8c5b"}
2024-06-24T13:52:29Z    INFO    Reconciling ScaledObject        {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"http-scaledobject-orderapplication","namespace":"daprpoc"}, "namespace": "daprpoc", "name": "http-scaledobject-orderapplication", "reconcileID": "7ae6e900-a5fe-4284-a0e8-22908be5ff63"}
2024-06-24T13:52:30Z    INFO    Reconciling ScaledObject        {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"azure-service-bus-scaledobject","namespace":"daprpoc"}, "namespace": "daprpoc", "name": "azure-service-bus-scaledobject", "reconcileID": "5ff2ebfd-8635-4b5f-944a-40cb72ec78a6"}
2024-06-24T13:52:30Z    INFO    Reconciling ScaledObject        {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"keda-add-ons-http-interceptor","namespace":"kube-system"}, "namespace": "kube-system", "name": "keda-add-ons-http-interceptor", "reconcileID": "f82dcec1-aa7b-422d-a864-89ccce837edb"}

[JorTurFer]

I'd like to see the error of the 1st pod: http-controller-manager. That's the add-on operator and the responsible of updating the status for HTTPScaledObjects, if there is any error with that resource, that's the pod for checking it :)

[mvanhil]

kubectl -n kube-system logs keda-add-ons-http-controller-manager-64dfdfcd7-fnfv4
Defaulted container "kube-rbac-proxy" out of: kube-rbac-proxy, keda-add-ons-http-operator
I0616 01:29:38.331628       1 main.go:186] Valid token audiences:
I0616 01:29:38.331963       1 main.go:316] Generating self signed cert as no cert is provided
I0616 01:29:40.032044       1 main.go:366] Starting TCP socket on 0.0.0.0:8443
I0616 01:29:40.032807       1 main.go:373] Listening securely on 0.0.0.0:8443

[JorTurFer]

Mb, I need the other container (keda-add-ons-http-operator)

[mvanhil]

Sorry, I don't have that pod. FYI: I run on AKS.