AWS::Http::CurlHandleContainer will cause the service to be permanently unavailable

[USTCZJY]

Describe the bug

The DestroyCurlHandle function will fail to rebuild the curl connection, but at this time m_poolSize has not changed, which may result in no connection available in the end. For example: there are N connections in the connection pool, and there are currently 2N concurrent requests. Unfortunately, the N threads that have obtained the connection have problems when making http calls, and DestroyCurlHandle fails to rebuild the curl connection. At this time, there is no connection in the connection pool, but m_poolSize is equal to m_maxPoolSize, so the blocked N threads cannot be awakened, and the new working thread will also be blocked, causing the service to be permanently unavailable.
2. If function AWS::Utils::ExclusiveOwnershipResourceManager::ShutdownAndWait encounters the above situation, it will also be permanently blocked because it does not meet the condition of _resources.size() == resourceCount.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

none

Current Behavior

I haven't actually encountered it, but it's a potential risk.

Reproduction Steps

none

Possible Solution

none

Additional Information/Context

none

AWS CPP SDK version used

1.11.438

Compiler and Version used

4.8.5

Operating System and version

linux

[jmklix]

Do you have any potential repro code that might reproduce this? I noticed that you said you haven't encountered this. How likely do you expect this edge case to happen and how do you expect the sdk to handle it?

[USTCZJY]

The problem is mainly in this part of the code. When the curl_easy_init() function returns nullptr, the connection pool will permanently lack a connection, but m_poolSize will not change. This makes the connection pool actually have only N-1 connections, but the worker thread thinks the connection pool is full and will not call the grow operation. If this error continues to occur, the service will be unavailable. However, the probability of curl_easy_init() function returning nullptr error is mainly related to the actual operating environment and is difficult to estimate.

[USTCZJY]

As for the improvement method, I think you can follow the modification method in the figure, but this method still has problems, which will also happen when the connection reconstruction fails. For example: the connection pool has N connections, and there are N+1 worker threads to obtain connections, but the N threads that obtain the connection http requests all fail and the connection reconstruction fails. At this time, although m_poolSize is restored to 0, if there is no new worker thread to perform the grow operation, the previously blocked thread will never be awakened.

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.