-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Issue with AWS MSK IAM using Apache Kafka scaler #5531
Comments
Hi @sameerjoshinice, Thanks for reporting this to us. An i/o timeout and context deadline exceed often mean network connection error. I have a few questions:
|
Hi @dttung2905 , |
We also saw this error from the Keda operator before the timeouts and context deadline started happening: ERROR scale_handler error getting metric for trigger {"scaledObject.Namespace": "mynamespace", "scaledObject.Name": "myscaler", "trigger": "apacheKafkaScaler", "error": "error listing consumer group offset: %!w()"} |
Our suspicion is that the scaler caused a flood of broken connections that didn't close properly and eventually caused all of the groups to rebalance which pegged the CPU. The rebalances can be seen within a few minutes of starting the scalingobject. I also have this email which highlights some things AWS was finding at the same time: I’ve been talking to our AWS TAM and the AWS folks about this issue. They still believe based on the logs that they have access to (which we don’t) that the problems are related to a new IAM permission that is required when running under the newest Kafka version. They are seeing many authentication issues related to the routing pods. My coworker and I have been playing with permissions to give the application rights requested by AWS. The CPU on the cluster dropped slightly when we did that, however, we are getting the following error still even after applying the update on the routing pods: Connection to node -2 () terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue. AWS believes that the authentication sessions on the backend have been marked as expired, but they have not been removed and are treated as invalid. They have been attempting to manually remove them, but have run into problems doing so. They are going to restart all the MSK brokers to clear out the session cache. |
Unfortunately, restarting the brokers didn't fix the CPU problems. |
Did you try restarting KEDA operator? keda/pkg/scalers/apache_kafka_scaler.go Lines 564 to 574 in 367fcd3
But maybe there is any other way to close the client that we've missed :/ |
Following is the analysis from AWS MSK team for the issue. They see this as a problem in Keda scaler. The issue is mainly new apache kafka scaler keep on retrying constantly with non renewed credentials even after session expiry. Based on the authorizer logs, we see that KEDA is denied to access certain resources. This is leading to the same scaler retrying. This retry happens constantly until the session expires. When session expires, the credential is not renewed by KEDA, and thus, it attempts to call the cluster with an outdated credential. This leads to a race condition where the requests are constantly in AUTHENTICATION failed state. This leads to the request queue, and then the network queue filling up, which leads to high CPU. The behaviour on MSK side is that the failure is happening in the authorization flow, and not the authentication flow. This was hard to reproduce. However, when we add latency in between the server and authorization flow which might happen in case of network queue being overloaded, we can reproduce this behaviour. The rejection of these requests is correct. Under normal circumstances, they would be rejected in authentication flow, but under heavy load, they are being rejected in authorization flow. Mitigation An issue should be raised with KEDA about this. The scaler will always eventually crash if authentication or authorization fails. This can trigger with any KEDA scaler if permissions are not sufficient. It will keep retrying until session expires, and then face this issue. |
Been a while since I worked with Kafka, I made the initial commit for the scaler. If it's an issue with rotating credentials I think it would be better to raise the issue in over at segmentio/kafka-go. They are the ones maintaining the underlying library. Could it be confirmed if it's the newer versions of Kafka having the issue? Or for that matter if earlier builds of Keda have the issue. I don't see many changes in in this repo that would give this issue. |
Is this specific to RBAC or same problem can be seen when an IAM user is used? I don't have access to infra to test this on tbh. |
Version of Kafka being used is 3.5.1. We could not confirm if it was an issue with earlier version or not. |
|
@dttung2905 @zroubalik , do you know how the integration of AWS MSK IAM within sarama is? I mean, if we are close to unify both scalers again we can just work in that direction |
I see AWS has released the signer https://github.com/aws/aws-msk-iam-sasl-signer-go . |
Yeah, we knew it because another folk told it to use some weeks ago, IIRC @dttung2905 is checking how to integrate it |
@sameerjoshinice yes. I think thats the repo to use. It has been mentioned in this Sarama issue IBM/sarama#1985 @JorTurFer I did not recall I was looking into that sorry. But I'm happy to test it out if I could get my hands on a testing MSK environment :D |
Lol, maybe I'm wrong but that what's I remember 😰 xD No worries, IDK why I remembered it but surely I was wrong |
👋 Greetings! Would you be open if I were to look at this issue and see how we could solve this with integrating the signer from AWS? |
Yeah, if you are willing to give a try, it'd be nice! |
@adrien-f thanks, let me assing this issue to you. |
Greetings 👋 I was able to get our MSK cluster (1000+ topics) to enable IAM authentication. With the current codebase, it connected fine so that's good news on the current state of the authentication system. Immediately, I notice the scaler runs the following: keda/pkg/scalers/apache_kafka_scaler.go Lines 424 to 431 in 80806a7
That MetadataRequest is not scoped to the topic the scaler is looking at. Which means it retrieves all information for all topics & partitions on the cluster and it can get big. I think it could be fair to already scope that data to the topic configured for the scaler. Moreover, the following is from the Kafka protocol documentation:
Caching topic partitions might also be another option to investigate. I will continue investigating and open a PR with these suggestions 👍 |
@adrien-f I thought we were discussing the idea of modifying the existing Kafka scaler based on sarama library than fixing the issue with the experimental scaler based on segment-io library. |
Hey @sameerjoshinice ! Isn't |
https://keda.sh/docs/2.13/scalers/apache-kafka/ the first one |
@adrien-f |
Got it 👍 I'll look at adding that ! |
Hey there! I've implemented the MSK signer! Let me know what you think :) |
I think part of the issue might be related to #5806. |
This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions. |
This issue has been automatically closed due to inactivity. |
Report
AWS MSK getting into high CPU usage and retrieval of metadata not working for Apache Kafka scaler experimental
Expected Behavior
After having everything correctly configured, Keda should have been able to get the metadata for the topics, use it for scaling and not affect MSK itself.
Actual Behavior
No metadata retrieval working giving errors, causing high CPU usage on MSK causing MSK outage. This means scaler is not working as expected.
Steps to Reproduce the Problem
2.Sasl is set to aws_msk_iam and tls is set to enable.
Logs from KEDA operator
KEDA Version
2.13.0
Kubernetes Version
1.26
Platform
Amazon Web Services
Scaler Details
Apache Kafka scaler (experimental)
Anything else?
This caused a major outage for us since we use shared MSK. This is a big problem for other services that got affected because of this scaler. Even after restart of brokers, the issue remains because Kafka keeps the information about these connections and is taking lot of time to stabilize after that.
The text was updated successfully, but these errors were encountered: