Replies: 1 comment
-
We noticed a high number of idle connections and decided to exclude redis traffic from Envoy on the client side. The number of idle connections immediately dropped. Both performance and the number of connections across all leaders is much more even now.
|
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Using the ClusterClient, we're seeing an extremely high number of connections opened to Redis. In the screenshot below, half the requests are MSets and half are MGets. All commands are wrapped in pipelines and we do not use hashtags. We have 15 nodes in the cluster. Usually both the client and Redis are inside an istio mesh, but we have pulled Redis outside the mesh for the purposes of this test. This is on a GKE cluster.
There's certainly additional overhead in clustered redis as go-redis breaks up a pipeline by node, so some fan-out is expected. However, there's an overwhelming amount of connections being opened to Redis, to the point the majority of our read requests are taking > 15ms and start to fail due to i/o timeout.
ClusterClient config immediately before client is created
Interestingly the clients specified in the original config have significantly higher open connections. IIUC the client should be updated with all leaders in the cluster and round-robin between them
Example trace
Beta Was this translation helpful? Give feedback.
All reactions