KEDA 2.6.1 Unable to get external metric for AWS SQS. #2632
Comments
|
hi @vujadeyoon |
|
Hi, @JorTurFer I attach the logs for both pods you mentioned to the above Section. Logs from KEDA operator.
Please note that I checked the ${MY_OWN_AWS_ACCESS_KEY_ID} is correct (i.e. the value is my AWS_ACCESS_KEY_ID). I just change the own value to ${MY_OWN_AWS_ACCESS_KEY_ID} in this markdown file. Best, |
|
hi @vujadeyoon I think that the problem could be a wrong configuration |
|
try removing this line |
|
Dear @JorTurFer , I managed to solve the problem right now! I also requested the AWS support because the AWS also recommend to use KEDA officially [1]. I attach all commands to attach the policy to the corresponding worker node that has a KEDA operator pod as below. # ${NAME_STACK_NODEGROUP}: This value can be checked in the AWS CloudFormation.
$ aws_account_id=$(aws sts get-caller-identity --query Account --output text)
$ name_role=$(aws cloudformation describe-stack-resource --stack-name ${NAME_STACK_NODEGROUP} --logical-resource-id NodeInstanceRole --query 'StackResourceDetail' | jq -r '.PhysicalResourceId')
$ name_policy_arn=arn:aws:iam::${aws_account_id}:policy/AWSSQSGetQueueAttrIAMPolicy
$ aws iam attach-role-policy --role-name ${name_role} --policy-arn ${name_policy_arn}Finally, I really appreciate @JorTurFer because of giving me advice to solve this problem. Thanks to you, I could notice the important things related to the IAM policy. Best reagrds, [1] https://github.com/amazon-archives/k8s-cloudwatch-adapter {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sqs:GetQueueAttributes",
"Resource": "<AWS_SQS_QUEUE_ARN>" # You can get the ARN value in the your AWS SQS queue.
}
]
} |
|
@vujadeyoon Can you share the steps on how you have solved your issue. |
|
Dear @Hema-Waters I wrote the way [1] to solve the issue above, as far as you can see. I recommend you should follow the above my reply. If you have additional questions, please give me details. Best regards, [1] #2632 (comment) |
Report
Dear all,
I try to use the KEDA 2.6.1 for AWS SQS [1]. (I also tried other KEDA versions like 2.4.0, 2.5.0 and 2.6.0, but I got same results form the all KEDA versions.)
After deploying the KEDA 2.6.1 and applying the keda-resource.yaml [2], I got some error as below.
unable to get external metric ns-vujade/s0-aws-sqs-eks-sqs-fifo/&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name:You can check it in details in [3].
I also checked previous issues, #868 and #2381, but it can't solve my problem.
Please feel free to contact me via this channel to solve and debug this issue.
Best regards,
Vujadeyoon
[1] https://keda.sh/docs/2.6/scalers/aws-sqs/#scaling-a-deployment-using-iam-users
[2] keda-resource.yaml
[3] HPA results
Name: keda-hpa-aws-sqs-queue-scaledobject Namespace: ns-vujade Labels: app.kubernetes.io/managed-by=keda-operator app.kubernetes.io/name=keda-hpa-aws-sqs-queue-scaledobject app.kubernetes.io/part-of=aws-sqs-queue-scaledobject app.kubernetes.io/version=2.6.1 scaledobject.keda.sh/name=aws-sqs-queue-scaledobject Annotations: <none> CreationTimestamp: Mon, 14 Feb 2022 21:44:39 +0900 Reference: Deployment/nginx-deployment Metrics: ( current / target ) "s0-aws-sqs-eks-sqs-fifo" (target average value): <unknown> / 2 Min replicas: 3 Max replicas: 36 Deployment pods: 3 current / 3 desired Conditions: Type Status Reason Message ---- ------ ------ ------- AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale ScalingActive False FailedGetExternalMetric the HPA was unable to compute the replica count: unable to get external metric ns-vujade/s0-aws-sqs-eks-sqs-fifo/&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: aws-sqs-queue-scaledobject,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: no matching metrics found for s0-aws-sqs-eks-sqs-fifo Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal SuccessfulRescale 9m40s horizontal-pod-autoscaler New size: 3; reason: Current number of replicas below Spec.MinReplicas Warning FailedComputeMetricsReplicas 6m38s (x12 over 9m24s) horizontal-pod-autoscaler invalid metrics (1 invalid out of 1), first error is: failed to get s0-aws-sqs-eks-sqs-fifo external metric: unable to get external metric ns-vujade/s0-aws-sqs-eks-sqs-fifo/&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: aws-sqs-queue-scaledobject,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: no matching metrics found for s0-aws-sqs-eks-sqs-fifo Warning FailedGetExternalMetric 4m37s (x20 over 9m24s) horizontal-pod-autoscaler unable to get external metric ns-vujade/s0-aws-sqs-eks-sqs-fifo/&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: aws-sqs-queue-scaledobject,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: no matching metrics found for s0-aws-sqs-eks-sqs-fifoExpected Behavior
The error messages which are shown in a following command should not be displayed.
Then, the HPA can work (i.e. scale-out and scale-in) as the queue size in the AWS SQS is increased or decreased.
Actual Behavior
As far as I mentioned above Report section, the actual result (i.e. [3] HPA result) is as below.
Please note that the HPA also doesn't work.
[3] HPA results
Name: keda-hpa-aws-sqs-queue-scaledobject Namespace: ns-vujade Labels: app.kubernetes.io/managed-by=keda-operator app.kubernetes.io/name=keda-hpa-aws-sqs-queue-scaledobject app.kubernetes.io/part-of=aws-sqs-queue-scaledobject app.kubernetes.io/version=2.6.1 scaledobject.keda.sh/name=aws-sqs-queue-scaledobject Annotations: <none> CreationTimestamp: Mon, 14 Feb 2022 21:44:39 +0900 Reference: Deployment/nginx-deployment Metrics: ( current / target ) "s0-aws-sqs-eks-sqs-fifo" (target average value): <unknown> / 2 Min replicas: 3 Max replicas: 36 Deployment pods: 3 current / 3 desired Conditions: Type Status Reason Message ---- ------ ------ ------- AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale ScalingActive False FailedGetExternalMetric the HPA was unable to compute the replica count: unable to get external metric ns-vujade/s0-aws-sqs-eks-sqs-fifo/&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: aws-sqs-queue-scaledobject,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: no matching metrics found for s0-aws-sqs-eks-sqs-fifo Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal SuccessfulRescale 9m40s horizontal-pod-autoscaler New size: 3; reason: Current number of replicas below Spec.MinReplicas Warning FailedComputeMetricsReplicas 6m38s (x12 over 9m24s) horizontal-pod-autoscaler invalid metrics (1 invalid out of 1), first error is: failed to get s0-aws-sqs-eks-sqs-fifo external metric: unable to get external metric ns-vujade/s0-aws-sqs-eks-sqs-fifo/&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: aws-sqs-queue-scaledobject,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: no matching metrics found for s0-aws-sqs-eks-sqs-fifo Warning FailedGetExternalMetric 4m37s (x20 over 9m24s) horizontal-pod-autoscaler unable to get external metric ns-vujade/s0-aws-sqs-eks-sqs-fifo/&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: aws-sqs-queue-scaledobject,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: no matching metrics found for s0-aws-sqs-eks-sqs-fifoSteps to Reproduce the Problem
I attach all commands that I executed.
Please note that I don't set any other IAM role and policy because the official guide does't include handling a IAM role and policy. I just only made a queue (i.e. queue.fifo) in the AWS SQS before executing below commands.
$ wget https://github.com/kedacore/keda/releases/download/v2.6.1/keda-2.6.1.yaml $ kubectl apply -f keda-2.6.1.yaml $ kubectl create ns ns-vujade $ kubectl apply -f kube-resource.yaml # The kube-resource.yaml [2] is attached in the above Report section. $ kubectl describe hpa -ALogs from KEDA operator
I attach some logs for referencing the previous issue, #2381.
{ "kind": "APIResourceList", "apiVersion": "v1", "groupVersion": "external.metrics.k8s.io/v1beta1", "resources": [ { "name": "s0-aws-sqs-eks-sqs-fifo", "singularName": "", "namespaced": true, "kind": "ExternalMetricValueList", "verbs": [ "get" ] } ] }"s0-aws-sqs-eks-sqs-fifo"KEDA Version
2.6.1
Kubernetes Version
1.21
Platform
Amazon Web Services
Scaler Details
AWS SQS
Anything else?
I got same results (i.e. same errors) regardless less of the KEDA versions as far as I mentioned in the above Report section.
The text was updated successfully, but these errors were encountered: