-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Application does not recover following loss of connectivity with DB #106
Application does not recover following loss of connectivity with DB #106
Comments
crystal-db support reconnection natively. #107 also benefits from the information above |
So as long as this statement_with_retry is used https://github.com/crystal-lang/crystal-db/blob/bf5ca75d1ace7e15b00ca03ad21728b8b00cf007/src/db/pool_statement.cr#L41-L45 like here https://github.com/crystal-lang/crystal-db/blob/bf5ca75d1ace7e15b00ca03ad21728b8b00cf007/src/db/pool_statement.cr#L23-L25 then ConnectionLost won't be raised def retry in statement_with_retry is defined here https://github.com/crystal-lang/crystal-db/blob/bf5ca75d1ace7e15b00ca03ad21728b8b00cf007/src/db/pool.cr#L179 Basically it rescues PoolResourceLost(T) errors (ConnectionLost < PoolResourceLost(Connection)), the only error statement_with_retry would not rescue is PoolRetryAttemptsExceeded which happens if retry_attempts + current connections is too low. So what this means is somewhere in crystal-pg or clear, the query method that is implemented does not use statement_with_retry and execute the sql code directly. If we look at https://github.com/will/crystal-pg/blob/cafe572e12966f7449477891fa65558954dbca5c/src/pg/statement.cr#L1-L44, which is used throughout clear (defined in db: https://github.com/crystal-lang/crystal-db/blob/bf5ca75d1ace7e15b00ca03ad21728b8b00cf007/src/db/statement.cr#L63-L110) We can clearly see that there is no retry logic here. We can implement the retry logic like https://github.com/crystal-lang/crystal-db/blob/bf5ca75d1ace7e15b00ca03ad21728b8b00cf007/src/db/pool.cr#L165-L191 but we need access to @retry_attempts, @retry_delay, and @idle, however those are only available in the recovery testing steps:
|
Relevant: crystal-lang/crystal-db#45 (comment) |
Hi! I'm trying to understand the context of the issue reported in crystal-db a little bit. 👀 I think that place-labs/clear#11 will not recover from connection loss. Once your are dealing with a connection object, there is no retry logic possible. A connection is almost a socket. If the socket is broken/closed you are done. But if you run statements using the pool then the pool can recover. Because you don't care about the which connection is actually used. (This also implies like transactions can't recover. At least there is no API to retry the whole transaction if the connection goes down). So, if |
Describe the bug
Following an interruption to database availability, the application can enter a state where requests fail and recovery becomes dependent on infrastructure automation.
Your environment
placeos-1.2108.4
Expected behavior
Reconnection should be attempted.
Additional context
Example log output under failure conditions:
The ORM in use appears to support this, but is disabled by default. If this is compatible, it would be good to enable this behaviour by default. Alternatively, if the upstream behaviour is not suitable, an alternative application level recover should be added here.
The text was updated successfully, but these errors were encountered: