-
Notifications
You must be signed in to change notification settings - Fork 817
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Panic in (*Pool).SendBatch: close of closed channel #1920
Comments
That path is only hit when the PostgreSQL server returns a |
@jackc I will add to what I mentioned above that after further investigation we found that this happened upon our Postgres DB instance being restarted as part of a maintenance by GCP (as it's managed by GCP).
In addition, as I said we are using the same pool object concurrently from several go-routines sharing a pointer to it. Is it possible that this occurred as part of some unresolved race condition that might happen if a |
Otherwise, it might be possible to panic when closing the pipeline if it tries to read a connection that should be closed but still has a fatal error on the wire. #1920
I think I have resolved the issue in 2e84dcc, but with something so rare it is hard to be sure. It appears that the problem could occur in pipeline mode (which is used by a batch) if the connection was killed while reading the responses. The closing the pipeline would cause it to read the remaining expected responses. But if the connection was dead, but somehow there was still a FATAL error on the wire it could trigger the problem. I was only able to induce this by mocking a PostgreSQL server and purposed sending two FATAL error messages in a row. To my knowledge the real PostgreSQL server never does this. But perhaps PgBouncer does... Anyway, regardless of exactly how that state is reached, I think this should fix it. |
@jackc Thanks, how can we use a version of the library with this fix? Is there a new release of the library planned and if so can you tell us when we can expect it to be released? |
The next patch release will probably be some time next week. |
Describe the bug
We have a service in Go that uses
*Pool
fromvendor/github.com/jackc/pgx/v5/pgxpool/pool.go
to perform write statements (insert/update/delete) on tables in a Postgres instance (technically the connection is directly established to pgbouncer instance and the pgbouncer instance connects to the DB instance itself, but I'm not sure how relevant it is).The write statements are performed using the pgx
Batch
construct and the(*Pool).SendBatch
method.We use version v5.4.3 of pgx, use Go version 1.21.5 and our Postgres DB instance runs version 15.4.
We have several hundred instances of the service running with roughly the same code in this area for at least 3 months connected to several different Postgres instances indirectly via pgbouncer and today for the first time one of the instances of our service encountered a panic with the following stack trace:
The part omitted after the ellipsis is our application code, the first call to the pgx package's code is the call to
(*Pool).SendBatch
.To Reproduce
How we initialize the client:
A schematic example of how we create a batch, add a query to the batch and invoke the client's
SendBatch
method:Several invocations of similar versions of the above
update()
function run concurrently in separate go-routines where all go-routines share a pointer to the same instance of*Pool
.Expected behavior
No panic.
Actual behavior
A panic happens.
Version
Additional context
N/A
The text was updated successfully, but these errors were encountered: