proposal: os: (*File).SetReadDeadline and (*File).SetWriteDeadline

[ccbrown]

Proposal

Add three new methods to os.File:

// SetDeadline sets the read and write deadlines associated
// with the connection. It is equivalent to calling both
// SetReadDeadline and SetWriteDeadline.
//
// A deadline is an absolute time after which I/O operations
// fail with a timeout (see type Error) instead of
// blocking. The deadline applies to all future and pending
// I/O, not just the immediately following call to Read or
// Write. After a deadline has been exceeded, the connection
// can be refreshed by setting a deadline in the future.
//
// An idle timeout can be implemented by repeatedly extending
// the deadline after successful Read or Write calls.
//
// A zero value for t means I/O operations will not time out.
SetDeadline(t time.Time) error

// SetReadDeadline sets the deadline for future Read calls
// and any currently-blocked Read call.
// A zero value for t means Read will not time out.
SetReadDeadline(t time.Time) error

// SetWriteDeadline sets the deadline for future Write calls
// and any currently-blocked Write call.
// Even if write times out, it may return n > 0, indicating that
// some of the data was successfully written.
// A zero value for t means Write will not time out.
SetWriteDeadline(t time.Time) error

If these look familiar, it's because I've copied and pasted them from the net package's Conn interface.

In Go 1.9, the os package now uses the internal runtime poller for file IO where possible (release note). Internally, there is already good support for timing out reads and writes on pollable files (The poll file descriptor already has these methods.). So the amount of engineering effort required to implement this proposal should be relatively minimal for those. Files that are not pollable can just return an error if you try to set a deadline (in the same manner as the net package's pipes do). See also:

• os: support runtime poller with os.File on Solaris
• os: support runtime poller with os.File on Windows

An Alternative Proposal

Add two new methods to os.File:

ReadContext(ctx context.Context, b []byte) (n int, err error)
WriteContext(ctx context.Context, b []byte) (n int, err error)

Again, these would be only supported by pollable files, and would immediately return an error for all other files.

[davecheney]

Thank you for raising this proposal. How does this proposal address the issue of a file descriptor being reused if Close is called concurrently with Read/Write that you mention in the introduction?

…

On Mon, Sep 11, 2017 at 6:10 PM, Chris ***@***.***> wrote: *Background* There's currently no well-supported way to safely abort or time out a Read or Write on an os.File. Unfortunately, many people end up doing this by invoking Close from another goroutine, which is not well-defined behavior and potentially very unsafe if the closed file descriptor is re-used by the kernel at the wrong time. Some discussion on the matter: - os: race between os.(*File).Close() and os.(*File).Read() <#10001> - proposal: "os/poll" package for exposing the runtime poller for generic file descriptors <#18507> - How do I cancel a file copy without a race? <https://groups.google.com/forum/#!topic/golang-nuts/i4w58KJ5-J8> - SetReadDeadline for os.File? <https://groups.google.com/forum/#!topic/golang-nuts/uTsVAkDLINI> The only safe way to abort or time out a Read or Write is to use syscall. But that also presents challenges such as implementing FD_SET and friends. Some more discussion on the matter specifically from the syscall.Select angle: - syscall.Select <#500> - Poll and Select syscalls <https://groups.google.com/forum/#!topic/golang-nuts/al-6_QFgYaM> - syscall.Read() with timeout: syscall.Select() vs golang's select statement <https://groups.google.com/forum/#!msg/golang-nuts/Wdpj20r360A/2gvP9q6hCwAJ> - fd polling with syscall.Select <http://grokbase.com/t/gg/golang-nuts/14at84gaby/go-nuts-fd-polling-with-syscall-select> Clearly aborting or timing out reads and writes is something that people both want to do and have a lot of trouble doing correctly. *Proposal* Add three new methods to os.File: // SetDeadline sets the read and write deadlines associated// with the connection. It is equivalent to calling both// SetReadDeadline and SetWriteDeadline.//// A deadline is an absolute time after which I/O operations// fail with a timeout (see type Error) instead of// blocking. The deadline applies to all future and pending// I/O, not just the immediately following call to Read or// Write. After a deadline has been exceeded, the connection// can be refreshed by setting a deadline in the future.//// An idle timeout can be implemented by repeatedly extending// the deadline after successful Read or Write calls.//// A zero value for t means I/O operations will not time out.SetDeadline(t time.Time) error // SetReadDeadline sets the deadline for future Read calls// and any currently-blocked Read call.// A zero value for t means Read will not time out.SetReadDeadline(t time.Time) error // SetWriteDeadline sets the deadline for future Write calls// and any currently-blocked Write call.// Even if write times out, it may return n > 0, indicating that// some of the data was successfully written.// A zero value for t means Write will not time out.SetWriteDeadline(t time.Time) error If these look familiar, it's because I've copied and pasted them from the net package's Conn <https://golang.org/pkg/net/#Conn> interface. In Go 1.9, the os package now uses the internal runtime poller for file IO where possible (release note <https://golang.org/doc/go1.9#os>). Internally, there is already good support for timing out reads and writes on pollable files (The poll file descriptor already has these methods. <https://golang.org/pkg/internal/poll/#FD.SetDeadline>). So the amount of engineering effort required to implement this proposal should be relatively minimal for those. Files that are not pollable can just return an error if you try to set a deadline (in the same manner as the net package's pipes do). See also: - os: support runtime poller with os.File on Solaris <#19111> - os: support runtime poller with os.File on Windows <#19098> *An Alternative Proposal* Add two new methods to os.File: ReadContext(ctx context.Context, b []byte) (n int, err error)WriteContext(ctx context.Context, b []byte) (n int, err error) These would enable users to abort a read or write with no wait, but the deadline methods might be more consistent with existing io.ReadWriter implementations such as net.Conn. Again, these would be only supported by pollable files, and would immediately return an error for all other files. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#21834>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAcA3mamhUJHM3kbd-9ZP7g7Mf11lTeks5shOrfgaJpZM4PSyUi> .

[ccbrown]

@davecheney It would give you a safe alternative to calling Close concurrently. Instead, you would wait until your Read or Write is unblocked by a timeout (or context cancellation in the case of the alternative proposal), then Close the file.

[davecheney]

I don't see how this proposal closes the read/close fd reuse hole. I would prefer that assertion was removed from this proposal. Thanks.

…

On Mon, Sep 11, 2017 at 6:17 PM, Chris ***@***.***> wrote: @davecheney <https://github.com/davecheney> It would give you a safe alternative to calling Close concurrently. Instead, you would wait until your Read or Write is unblocked by a timeout (or context cancellation in the case of the alternative proposal), then Close the file. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#21834 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAcA3qrtYY2iyZdAXErr37jqABeMUoSks5shOywgaJpZM4PSyUi> .

[dsnet]

Related to #20280.

\cc @zombiezen

[ccbrown]

I don't see how this proposal closes the read/close fd reuse hole. I would
prefer that assertion was removed from this proposal. Thanks.

@davecheney I'll gladly remove any incorrect or irrelevant assertions from my proposal. But I'm not sure what you're asking me to remove. I'm definitely not asserting that this proposal closes the file descriptor re-use hole. The context about the file descriptors is there because it reinforces the premise that I'm using to justify the proposal:

Aborting or timing out reads and writes is something that people both want to do and have a lot of trouble doing correctly.

The first set of links is to help illustrate the need: This proposal would provide an elegant solution to all of those posters. The second set of links is to help illustrate that it's currently very difficult to accomplish this safely: Implementing your own solution via syscall isn't portable or well-supported at all.

[ianlancetaylor]

In Go 1.9 and later the os package will protect against calling Close in one goroutine while there is a Read or Write occurring in another goroutine. The file descriptor will not be closed until the Read or Write completes.

This proposal is still meaningful but I believe the background section is simply wrong. I suggest editing the proposal to drop the background section, or closing this one and starting over. Sorry.

[ccbrown]

@ianlancetaylor I'm aware of the 1.9 behavior, but that doesn't invalidate my points. Even if the 1.9 implementation for some platforms behaves well, it's definitely not "well-supported" or "safe" (words that I was very careful to qualify my statements with) to rely on this implementation detail.

I think the background section is far from "simply wrong" and is extremely relevant, but I'll remove it from the proposal since for whatever reason it seems to be a distraction. People can draw their own conclusions about why os.File cancellation or timeouts may or may not be useful.

[ianlancetaylor]

@ccbrown My point was that in 1.9 calling Close in one goroutine while another goroutine is calling Read or Write is indeed safe. It is not an implementation detail. Before 1.9 it was a bug that this case was broken (#7970), and now that bug is fixed.

[ccbrown]

@ianlancetaylor I see. If it's the case that the behavior of Close during Read or Write is well-defined and safe on all platforms, I'm gonna close this issue. That satisfies my use-case as well as the use-cases I previously referenced. Thanks for your time!

[juliandroid]

Basically, we don't have solution for non-blocking reads (os.File / Read()) and the only way to exit a goroutine is by closing the file (which is needed, in my case, at other places/goroutines and is not acceptable to be closed)?

I guess the current workaround is to use syscall.O_ASYNC|syscall.O_NONBLOCK and syscall.Read

[ianlancetaylor]

@juliandroid This issue is closed, as it was more or less focused on safely using concurrent Read/Write and Close. If you want to open a new issue, please do that. If you want to discuss the matter, please use golang-dev. Thanks.