Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(pgwire): allow configuring tcp keepalive idle time #18136

Merged
merged 4 commits into from
Aug 22, 2024

Conversation

BugenZhao
Copy link
Member

@BugenZhao BugenZhao commented Aug 20, 2024

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

It's common for an SQL statement not returning any data from the pgwire server for a long time (typically when executing DDL). If the connection is proxied through a load balancer, the LB may consider the connection idle and drop it unexpectedly.

To resolve this issue, we may set the tcp keepalive idle time to a lower value. Here we take a reference to the value in AWS ELB, which is 350 seconds. Note that this can be done in any place dealing with the connection, but doing it on the pgwire server eliminates the need for any additional configuration.

The value is configurable through CLI args and defaults to 300 seconds.


FYI, the default keepalive idle time is decided by the operating system, typically 2 hours.

Checklist

  • I have written necessary rustdoc comments
  • I have added necessary unit tests and integration tests
  • All checks passed in ./risedev check (or alias, ./risedev c)

Documentation

  • My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

Signed-off-by: Bugen Zhao <i@bugenzhao.com>
@BugenZhao BugenZhao requested a review from a team as a code owner August 20, 2024 09:33
@BugenZhao BugenZhao requested a review from stdrc August 20, 2024 09:33
@github-actions github-actions bot added the type/fix Bug fix label Aug 20, 2024
Copy link
Contributor

@chenzl25 chenzl25 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. It seems to be related to the DBT EOF error.

Comment on lines 85 to 89
{
let r = socket2::SockRef::from(&stream);
let ka = socket2::TcpKeepalive::new().with_time(Duration::from_secs(300));
r.set_tcp_keepalive(&ka)?;
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting. Is this a hack or a common usage of socket2?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's the common usage and the only way for setting these extra parameters until tokio-rs/tokio#3082 is addressed.

Signed-off-by: Bugen Zhao <i@bugenzhao.com>
@stdrc
Copy link
Contributor

stdrc commented Aug 21, 2024

What about cp to 2.0?

Signed-off-by: Bugen Zhao <i@bugenzhao.com>
@BugenZhao BugenZhao changed the title fix(pgwire): set tcp keepalive idle time to a lower value fix(pgwire): allow configuring tcp keepalive idle time Aug 21, 2024
Copy link
Contributor

@arkbriar arkbriar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Signed-off-by: Bugen Zhao <i@bugenzhao.com>
@BugenZhao BugenZhao added this pull request to the merge queue Aug 22, 2024
Merged via the queue into main with commit fe56ce2 Aug 22, 2024
29 of 30 checks passed
@BugenZhao BugenZhao deleted the bz/pgwire-tcp-keep-alive branch August 22, 2024 04:19
github-actions bot pushed a commit that referenced this pull request Aug 22, 2024
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
github-merge-queue bot pushed a commit that referenced this pull request Aug 28, 2024
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants