Redis connection issue

[Zsombroo]

In the past weeks, there were multiple outage events on Production due to connection issues between dmart and redis. In the logs we are finding entries with the following message: "No connection available."
The issue occured on the 1.1.4 dmart version where redis version was 7.2.3. with 5.0.1 python package.

We've tried to reproduce the issue on DEV environment performing load tests and experienced the same connection issues.
Also tried 1.1.10 version of dmart and the same issues occured.
Please identify the root cause and provide fix ASAP

In our load test, this error came up at around 800 active users with 120 calls per sec.
This redis-py github issue might be connected to the issue we experience redis/redis-py#2995

dmart v1.1.10

• Calling the /public/excute/query/products endpoint through jmeter.

• Using the default redis' BlockingConnectionPool's max_connections parameter value of 20, we experienced the "No connection available" error when we had 75 calls per second.

• We increased the redis' BlockingConnectionPool's max_connections parameter to 2000. With 150 calls per second we did not experience the "No connection available" error anymore.

• With 2000 max_connections, we are receiving a "Too many open files" os level error. We are using the default ulimit value of 2048.

dmart v1.1.4

• With 2000 max_connections, the following command shows increasing amount of open file descriptors:
  lsof | grep python | grep "localhost:redis" | wc -l
  Last confirmed value was over 15_000.

• With 20 max_connections the following command shows stable amount of open file descriptors (240-290):
  lsof | grep python | grep "localhost:redis" | wc -l

• With 20 max_connections, the following command shows increasing amount of open file descriptors:
  lsof | grep python | wc -l
  Last confirmed value was over 13_000.

[RaniaAbualnadi]

Could you please provide additional details regarding the operating system version and Python version you're using?

[Zsombroo]

OS version:
Operating System: Red Hat Enterprise Linux 9.3 (Plow)
Kernel: Linux 5.14.0-284.18.1.el9_2.x86_64
Python 3.11.5

We have conducted further load tests. Please find below the results. Under heavier load, we have experienced "No connection available." error coming from redis-py package.
Using version 5.0.1 when the load is sustained over several minutes, the connections in the BlockingConnectionPool get stuck as mentioned in redis/redis-py#2995
These connections are not correctly released and instead we are seeing Timeout errors when the the redis-py client used in dmart tries to take a connection from the connection pool. These connections are stuck indefinitely and will not serve any request, therefore dmart has to be restarted to drop existing connections and build new ones.
Using version 5.0.3 with sustained load we still get the "No connection available." error, but our understanding is that we are receiving it by design. There is a timeout on the redis requests so when the system slows down under heavy load and the response times increase over this timeout threshold, redis-py returns with this exception. However after the load decreases to a normal level connections are released correctly, and ther are usable again. This PR redis/redis-py#2999 mentioned in the redis/redis-py#2995 thread seems to address the stuck connections issue.

We propose fixing redis version in the requirements to 5.0.3 to solve this issue

dmart v1.1.11 - redis 5.0.3 -> good (Gets "no connection error", but correctly releases the connections once the load decreases)
dmart v1.1.10 - redis 5.0.3 -> good (Gets "no connection error", but correctly releases the connections once the load decreases)
dmart v1.1.4 - redis 5.0.3 -> good (Gets "no connection error", but correctly releases the connections once the load decreases)
dmart v1.1.11 - redis 5.0.1 -> bad (connections get stuck)
dmart v1.1.10 - redis 5.0.1 -> bad (connections get stuck)
dmart v1.1.4 - redis 5.0.1 -> bad (connections get stuck)

Edit 1:
Increasing the max_connections of the BlockingConnectionPool can delay the "No connection available." errors. It might be a good idea to add this value to the config.env file.

Edit 2:
Added dmart version 1.1.11 to the tests

[Zsombroo]

v1.1.12 fixes the issue