-
Notifications
You must be signed in to change notification settings - Fork 17.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
runtime: automatically bump RLIMIT_NOFILE on Unix #46279
Comments
The limitation on I note that on my Debian system the soft and hard limits are both |
Yeah, I saw We'd still need a conditional mechanism regardless for FWIW, on my various Debian (buster) & Ubuntu (focal LTS, hirsute) machines here, I see 1024 & 1048576. |
GitHub code search says https://github.com/search?l=&p=2&q=unix.Select+language%3AGo&type=Code .... it's mostly wireguard-go's (cc @zx2c4 as FYI) |
This proposal sounds like a good idea, with the caveat that we probably shouldn't do it in initialization for -buildmode=shared. |
What happens today, even in programs that do nothing but file I/O (no select etc), is that if you open too many files you get errors. Auto-bumping would let those programs run longer. If Go did it at startup, it would be inherited by non-Go programs that we fork+exec. That is a potential incompatibility, but probably not a large one. Technically, I suppose we could undo it in the subprocess between fork and exec. |
This proposal has been added to the active column of the proposals project |
To summarize the limitating use cases where we should not be raising the soft limit.
|
One problem with restoring the limit in exec is we won't know if the limit was intentionally changed by the program in the interim. What about programs that explicitly raise the limit and then exec today? We would be dropping it back down. It seems like if we are going to raise the limit, we should just do that, not try to put it back. I just ran into this problem with gofmt on my Mac, where the limit defaults to 256 (and gofmt was editing many files in parallel). I'd love for Go to raise the limit there too. How much does it really matter if we raise the limit for a subprocess? People can always set the hard limit if they want Go not to try to bump the soft limit up. |
It's pretty awful that the limit is breaking completely reasonable Go programs like gofmt -w. It's very hard to see any programs benefiting from this limit in practice anymore. |
I think that seems quite reasonable. We can even document this in |
Not sure anyone is using syscall.Select for fd's anyway. |
Based on the discussion above, this proposal seems like a likely accept. |
Should the title be updated to mention Unix or something instead of Linux? |
The considerations may be different on different Unix systems. On Linux the details are somewhat specific to systemd. It may well be appropriate to do this on macOS also, but I don't know what the tradeoffs are there. Why does macOS have a default low limit? |
From what I was able to find, that default goes back to the very first OS X release and probably even back to BSD. The constant is there. Of course, not doing that on macOS is not a deal-breaker but an annoyance. |
The only issue I am aware of that can arise if RLIMIT_NOFILE is set to a very high value is, some binaries (that may be executed from a Go program and thus inherit the limit) want to do something like this (pseudocode): for fd := 3; fd < getrlimit(RLIMIT_NOFILE); fd++ {
close(fd) // or set CLOEXEC flag
} For a specific example, Most probably this should not be an issue, since Docker also does a similar thing (moby/moby#38814) and since everyone seems to be using containers now, let's hope that issues like this are fixed (yet better, maybe some programs have even started using Also, this is surely not a showstopper to accept the proposal -- just something to keep in mind. |
No change in consensus, so accepted. 🎉 |
Backport issue(s) opened: #59063 (for 1.19), #59064 (for 1.20). Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases. |
As of Go 1.21 syscall.Setrlimit and syscall.prlimit can affect starting a new process, by restoring the original NOFILE rlimit. That is recorded locally in the syscall package, so just always call the syscall functions. For golang/go#46279 Change-Id: I2f3dafe5562a7dde1297bad6f5d34a80af5d620b Reviewed-on: https://go-review.googlesource.com/c/sys/+/476695 Run-TryBot: Ian Lance Taylor <iant@golang.org> Run-TryBot: Ian Lance Taylor <iant@google.com> Reviewed-by: Ian Lance Taylor <iant@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Bryan Mills <bcmills@google.com> Auto-Submit: Ian Lance Taylor <iant@google.com>
Change https://go.dev/cl/478659 mentions this issue: |
Change https://go.dev/cl/478660 mentions this issue: |
…ild process If we increased the NOFILE rlimit when starting the program, restore the original rlimit when forking a child process. In CL 393354 the os package was changed to raise the open file rlimit at program start. That code is not inherently tied to the os package. This CL moves it into the syscall package. This is a backport of CLs 476096 and 476097 from trunk. For #46279 Fixes #59064 Change-Id: Ib813de896de0a5d28fa2b29afdf414a89fbe7b2a Reviewed-on: https://go-review.googlesource.com/c/go/+/478659 Run-TryBot: Ian Lance Taylor <iant@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: David Chase <drchase@google.com> Reviewed-by: Michael Knyszek <mknyszek@google.com> Reviewed-by: Tobias Klauser <tobias.klauser@gmail.com>
…ild process If we increased the NOFILE rlimit when starting the program, restore the original rlimit when forking a child process. In CL 393354 the os package was changed to raise the open file rlimit at program start. That code is not inherently tied to the os package. This CL moves it into the syscall package. This is a backport of CLs 476096 and 476097 from trunk. For #46279 Fixes #59063 Change-Id: I9be6ecc52d4f82eb226907611aec9df808e1da84 Reviewed-on: https://go-review.googlesource.com/c/go/+/478660 TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Michael Knyszek <mknyszek@google.com> Run-TryBot: Ian Lance Taylor <iant@golang.org> Reviewed-by: David Chase <drchase@google.com> Reviewed-by: Tobias Klauser <tobias.klauser@gmail.com>
Change https://go.dev/cl/486576 mentions this issue: |
Now Go only sets the rlimit for the parent and any fork/exec'ed process gets the rlimit that was the default before fork/exec. Ref: golang/go#46279 This fix got backported to [Go 1.20.4](golang/go@ecf7e00) breaking Talos. Talos used to set rlimit in the [`SetRLimit`](https://github.com/siderolabs/talos/blob/v1.4.2/internal/app/machined/pkg/runtime/v1alpha1/v1alpha1_sequencer_tasks.go#L302) sequencer task. This means any process started by `wrapperd` gets the default Rlimit (1024). Fix this by explicitly setting `rlimit` in `wrapperd` before we drop any capabilities. Fixes: siderolabs#7198 Signed-off-by: Noel Georgi <git@frezbo.dev>
Now Go only sets the rlimit for the parent and any fork/exec'ed process gets the rlimit that was the default before fork/exec. Ref: golang/go#46279 This fix got backported to [Go 1.20.4](golang/go@ecf7e00) breaking Talos. Talos used to set rlimit in the [`SetRLimit`](https://github.com/siderolabs/talos/blob/v1.4.2/internal/app/machined/pkg/runtime/v1alpha1/v1alpha1_sequencer_tasks.go#L302) sequencer task. This means any process started by `wrapperd` gets the default Rlimit (1024). Fix this by explicitly setting `rlimit` in `wrapperd` before we drop any capabilities. Fixes: siderolabs#7198 Signed-off-by: Noel Georgi <git@frezbo.dev> (cherry picked from commit a2565f6)
As of https://go.dev/cl/476695 golang.org/x/sys/unix can call syscall.prlimit, so we need such a function in libgo. For golang/go#46279 Fixes golang/go#59712 Change-Id: I87ad6daaba68c188fb0abecb30f7d574db1f2600 Reviewed-on: https://go-review.googlesource.com/c/gofrontend/+/486576 Reviewed-by: Cherry Mui <cherryyz@google.com> Reviewed-by: Than McIntosh <thanm@google.com>
As of https://go.dev/cl/476695 golang.org/x/sys/unix can call syscall.prlimit, so we need such a function in libgo. For golang/go#46279 Fixes golang/go#59712 Reviewed-on: https://go-review.googlesource.com/c/gofrontend/+/486576
…ild process If we increased the NOFILE rlimit when starting the program, restore the original rlimit when forking a child process. In CL 393354 the os package was changed to raise the open file rlimit at program start. That code is not inherently tied to the os package. This CL moves it into the syscall package. This is a backport of CLs 476096 and 476097 from trunk. For golang#46279 Fixes golang#59064 Change-Id: Ib813de896de0a5d28fa2b29afdf414a89fbe7b2a Reviewed-on: https://go-review.googlesource.com/c/go/+/478659 Run-TryBot: Ian Lance Taylor <iant@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: David Chase <drchase@google.com> Reviewed-by: Michael Knyszek <mknyszek@google.com> Reviewed-by: Tobias Klauser <tobias.klauser@gmail.com>
…ild process If we increased the NOFILE rlimit when starting the program, restore the original rlimit when forking a child process. In CL 393354 the os package was changed to raise the open file rlimit at program start. That code is not inherently tied to the os package. This CL moves it into the syscall package. This is a backport of CLs 476096 and 476097 from trunk. For golang#46279 Fixes golang#59064 Change-Id: Ib813de896de0a5d28fa2b29afdf414a89fbe7b2a Reviewed-on: https://go-review.googlesource.com/c/go/+/478659 Run-TryBot: Ian Lance Taylor <iant@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: David Chase <drchase@google.com> Reviewed-by: Michael Knyszek <mknyszek@google.com> Reviewed-by: Tobias Klauser <tobias.klauser@gmail.com>
Stop managing limitnofile, Bookworm systemd default is soft 1024 and hard 524288. Further, golang >= 1.19 'os' package bumps soft to hard limit automatically (cfr golang/go#46279) For example thanos-query-frontend where we don't explicitly set limitnofile: root@titan2001:~# cat /proc/$(pgrep -f "thanos query-frontend")/limits | grep files Max open files 524288 524288 files root@titan2001:~# systemctl show thanos-query-frontend | grep -i limitnofile LimitNOFILE=524288 LimitNOFILESoft=1024 Bug: T346950 Change-Id: I3df68d63c66293e4425cb7e67670e44459c4d474
Change https://go.dev/cl/539435 mentions this issue: |
Tests in rlimit_test.go exist to test the behavior of automatically bumping RLIMIT_NOFILE on Unix implemented in rlimit.go (issue #46279), with darwin-specific behavior split out into rlimit_darwin.go and the rest left empty in rlimit_stub.go. Since the behavior happens only on Unix, it doesn't make sense to test it on other platforms. Copy rlimit.go's 'unix' build constraint to rlimit_test.go to accomplish that. Also simplify the build constraint in rlimit_stub.go while here, so that its maintenance is easier and it starts to match all non-darwin Unix GOOS values (previously, 'hurd' happened to be missed). In particular, this fixes a problem where TestOpenFileLimit was failing in some environments when testing the wasip1/wasm port. The RLIMIT_NOFILE bumping behavior isn't implemented there, so the test was testing the environment and not the Go project. Updates #46279. For #61116. Change-Id: Ic993f9cfc021d4cda4fe3d7fed8e2e180f78a2ca Reviewed-on: https://go-review.googlesource.com/c/go/+/539435 Reviewed-by: Johan Brandhorst-Satzkorn <johan.brandhorst@gmail.com> Reviewed-by: Bryan Mills <bcmills@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
Change https://go.dev/cl/540615 mentions this issue: |
… rlimit_test.go Tests in rlimit_test.go exist to test the behavior of automatically bumping RLIMIT_NOFILE on Unix implemented in rlimit.go (issue #46279), with darwin-specific behavior split out into rlimit_darwin.go and the rest left empty in rlimit_stub.go. Since the behavior happens only on Unix, it doesn't make sense to test it on other platforms. Copy rlimit.go's 'unix' build constraint to rlimit_test.go to accomplish that. Leave out the simplification of the build constraint in rlimit_stub.go so that this CL remains a test-only fix. In particular, this fixes a problem where TestOpenFileLimit was failing in some environments when testing the wasip1/wasm port. The RLIMIT_NOFILE bumping behavior isn't implemented there, so the test was testing the environment and not the Go project. Updates #46279. For #61116. Fixes #63994. Change-Id: Ic993f9cfc021d4cda4fe3d7fed8e2e180f78a2ca Cq-Include-Trybots: luci.golang.try:go1.21-wasip1-wasm_wasmtime Reviewed-on: https://go-review.googlesource.com/c/go/+/539435 Reviewed-by: Johan Brandhorst-Satzkorn <johan.brandhorst@gmail.com> Reviewed-by: Bryan Mills <bcmills@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org> (cherry picked from commit b7cbcf0) Reviewed-on: https://go-review.googlesource.com/c/go/+/540615 Reviewed-by: Heschi Kreinick <heschi@google.com> Auto-Submit: Heschi Kreinick <heschi@google.com>
I just read http://0pointer.net/blog/file-descriptor-limits.html which in a nutshell says:
select
select
usersselect
won't work.I realize that since Go doesn't use select, the Go runtime could automatically do this fd soft limit bumping on Linux.
We do have a Select wrapper at https://pkg.go.dev/golang.org/x/sys/unix#Select, though, so perhaps we could do the same thing we did for #42347 in 18510ae (https://go-review.googlesource.com/c/go/+/299671) and do the bumping conditionally based on whether the
unix.Select
func is in the binary. Orcgo
too, I suppose.I suspect many users are unaware of this 512K hard limit that's free to bump up to. I certainly was unaware. (I normally have to go in and manual tweak my systemd limits instead, usually in response to problems once I hit the limit...) I think fixing it automatically would help more users than it'd hurt. (I actually can't think how it'd hurt anybody?)
I don't think we need it as a backpressure mechanism. As the blog post mentions, memory limits are already that mechanism.
/cc @ianlancetaylor @aclements @rsc @randall77
The text was updated successfully, but these errors were encountered: