-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(spanner): long running transaction clean up - disabled #8177
Changes from 37 commits
7e7375e
77959b4
643678e
454e2a6
37997cb
a99ae01
acfa972
89b31c1
c073016
e0d1afc
327cd79
ee6cd46
6094f0b
9314761
f578fac
6aa6558
2f927c4
3897b35
7e52ce0
1c15ef5
9efe433
4e2d597
7a7986e
5f2d55a
ed12707
cbfe38c
f0bacdb
168438d
fb7c2d7
04b01a6
0359065
bc48a8a
ba90f75
750b911
2e26b54
b9e272e
057fcfb
1efbea6
1ee50f8
ab268ca
15b7af3
7910e1f
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -31,4 +31,87 @@ row, err := client.Single().ReadRow(ctx, "Users", | |
if err != nil { | ||
log.Fatal(err) | ||
} | ||
``` | ||
|
||
### Session Leak | ||
A `Client` object of the Client Library has a limit on the number of maximum sessions. For example the | ||
default value of `MaxOpened`, which is the maximum number of sessions allowed by the session pool in the | ||
Golang Client Library is 400. You can configure these values at the time of | ||
harshachinta marked this conversation as resolved.
Show resolved
Hide resolved
|
||
creating a `Client` by passing custom `SessionPoolConfig` as part of `ClientConfig`. When all the sessions are checked | ||
out of the session pool, every new transaction has to wait until a session is returned to the pool. | ||
If a session is never returned to the pool (hence causing a session leak), the transactions will have to wait | ||
indefinitely and your application will be blocked. | ||
|
||
#### Common Root Causes | ||
The most common reason for session leaks in the Golang client library are: | ||
1. Not stopping a `RowIterator` that is returned by `Query`, `Read` and other methods. Always use `RowIterator.Stop()` to ensure that the `RowIterator` is always closed. | ||
2. Not closing a `ReadOnlyTransaction` when you no longer need it. Always call `ReadOnlyTransaction.Close()` after use, to ensure that the `ReadOnlyTransaction` is always closed. | ||
3. Not closing a `BatchReadOnlyTransaction` when you no longer need it. Always call `BatchReadOnlyTransaction.Close()` after use, to ensure that the `BatchReadOnlyTransaction` is always closed. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. For a different PR: We should change the implementation of There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
|
||
As shown in the example below, the `txn.Close()` statement releases the session after it is complete. | ||
If you don't do `txn.Close()`, the session is not released back to the pool. The recommended way is to use `defer` as shown below, | ||
harshachinta marked this conversation as resolved.
Show resolved
Hide resolved
|
||
```go | ||
client, err := spanner.NewClient(ctx, "projects/P/instances/I/databases/D") | ||
if err != nil { | ||
log.Fatal(err) | ||
} | ||
txn := client.ReadOnlyTransaction() | ||
defer txn.Close() | ||
``` | ||
|
||
#### Debugging and Resolving Session Leaks | ||
|
||
##### Logging inactive transactions | ||
Enabled by default, the logging option shares warn logs when you have exhausted >95% of your session pool. | ||
harshachinta marked this conversation as resolved.
Show resolved
Hide resolved
|
||
This could mean two things, either you need to increase the max sessions in your session pool (as the number | ||
harshachinta marked this conversation as resolved.
Show resolved
Hide resolved
|
||
of queries run using the client side database object is greater than your session pool can serve) or you may | ||
harshachinta marked this conversation as resolved.
Show resolved
Hide resolved
|
||
have a session leak. To help debug which transactions may be causing this session leak, the logs will also contain stack traces of | ||
transactions which have been running longer than expected if `TrackSessionHandles` under `SessionPoolConfig` is enabled. | ||
|
||
```go | ||
sessionPoolConfig := spanner.SessionPoolConfig{ | ||
TrackSessionHandles: true, | ||
InactiveTransactionRemovalOptions: spanner.InactiveTransactionRemovalOptions{ | ||
ActionOnInactiveTransaction: spanner.Warn, | ||
}, | ||
} | ||
client, err := spanner.NewClientWithConfig( | ||
ctx, database, spanner.ClientConfig{SessionPoolConfig: sessionPoolConfig}, | ||
) | ||
if err != nil { | ||
log.Fatal(err) | ||
} | ||
defer client.Close() | ||
|
||
// Example Log message to warn presence of long running transactions | ||
// session <session-info> checked out of pool at <session-checkout-time> is long running due to possible session leak for goroutine | ||
// <Stack Trace of transaction> | ||
|
||
``` | ||
|
||
##### Automatically clean inactive transactions | ||
When the option to automatically clean inactive transactions is enabled, the client library will automatically spot | ||
harshachinta marked this conversation as resolved.
Show resolved
Hide resolved
|
||
problematic transactions that are running for extremely long periods of time (thus causing session leaks) and close them. | ||
harshachinta marked this conversation as resolved.
Show resolved
Hide resolved
|
||
The session will be removed from the pool and be replaced by a new session. To dig deeper into which transactions are being | ||
closed, you can check the logs to see the stack trace of the transactions which might be causing these leaks and further | ||
debug them. | ||
|
||
```go | ||
sessionPoolConfig := spanner.SessionPoolConfig{ | ||
TrackSessionHandles: true, | ||
InactiveTransactionRemovalOptions: spanner.InactiveTransactionRemovalOptions{ | ||
ActionOnInactiveTransaction: spanner.WarnAndClose, | ||
}, | ||
} | ||
client, err := spanner.NewClientWithConfig( | ||
ctx, database, spanner.ClientConfig{SessionPoolConfig: sessionPoolConfig}, | ||
) | ||
if err != nil { | ||
log.Fatal(err) | ||
} | ||
defer client.Close() | ||
|
||
// Example Log message for when transaction is recycled | ||
// session <session-info> checked out of pool at <session-checkout-time> is long running and will be removed due to possible session leak for goroutine | ||
// <Stack Trace of transaction> | ||
``` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@arpan14 @olavloite @rahul2393 @asthamohta
Can you please take a look at the golang documentation for session leaks and give your LGTM?