fs: Add file descriptor support to *File() funcs #3163

jwueller · 2015-10-03T00:28:43Z

This is a re-targeted submission of the archived PR #8522, implementing the archived issue #8471.

The original description follows.

These changes affect the following functions and their synchronous counterparts:

fs.readFile()
fs.writeFile()
fs.appendFile()

If the first parameter is a uint32, it is treated as a file descriptor. In all other cases, the original implementation is used to ensure backwards compatibility. File descriptor ownership is never taken from the user.

The documentation was adjusted to reflect these API changes. A note was added to make the user aware of file descriptor ownership and the conditions under which a file descriptor can be used by each of these functions.

Tests were extended to test for file descriptor parameters under the conditions noted in the relevant documentation.

jwueller · 2015-10-03T07:56:44Z

/cc @trevnorris @misterdjules

This one also includes a test for fs.writeFileSync(), where the original PR did not.

ChALkeR · 2015-10-04T06:53:19Z

lib/fs.js

@@ -101,6 +101,10 @@ function nullCheck(path, callback) {
  return true;
 }

+function isFd(path) {
+  return (path >>> 0) === path && path >= 0;


path >= 0 can never be false here.

You seem to be correct. This is probably residue from this discussion.

To clarify, if the user passes readFile('0xff'), would that be interpreted as a file name or file descriptor?

This implementation would treat '0xff' as a file name.

Will all values where typeof === 'number' be treated as a number, all other values will be coerced to a string?

The original implementation does not seem to coerce at all. It errors with TypeError: path must be a string instead.

Nevertheless, >>> should always produce a number type, so the strict equality test can only ever succeed for typeof input === 'number', making everything else go down the file name branch, even if that results in another TypeError caused by the path argument not being a string. This behavior remains unchanged.

I'm pondering how much coercion is allowed. e.g. should we throw on NaN? my instinct is yes. all non uint32 values should throw. Which is exactly what you have now. :)

I generally avoid coercion as much as possible. It usually leads to rather unclear behavior. Throwing on NaN sounds like it would be expected. There is no sensible default we can choose in that case.

ChALkeR · 2015-10-04T06:58:54Z

This is at least semver-minor, but actually looks more like a semver-major to me.
What if someone has done something like

var num = 1;
…
fs.writeFile(num++, …);

Also, the logic is not very clear:

fs.writeFile(1,…) tries to write to file descriptor 1.
fs.writeFile(-1,…) tries to write to file name '-1'.
fs.writeFile(4294967295,…) tries to write to file descriptor 4294967295.
fs.writeFile(4294967296,…) tries to write to file name '4294967296'.

Understandable behavior IMO would be to check if the file name is a string, and if it's not a string nor a valid file descriptor — throw an Error. But that change would be clearly a semver-major.

Not sure what the others think.

Note: I left aside the actual feature introduced by this PR as I have no opinion on that, and covered only the API changes.

jwueller · 2015-10-04T09:50:55Z

@ChALkeR First of all, thanks for reviewing this!

fs.writeFile(-1), etc. will actually fail with TypeError: path must be a string, so there does not seem to be a type coercion backwards compatibility problem here. Throwing a dedicated Error in case of neither a string or a plausible file descriptor uint would probably be helpful, though.

@trevnorris actually proposed the uint behavior here, as opposed to going the typeof route. Some additional discussion happened here. Thoughts?

Additionally, maybe @chrisdickinson could shed some light on the originally envisioned behavior as well (original issue link).

trevnorris · 2015-10-07T19:48:17Z

What are potential blockers here?

jwueller · 2015-10-08T10:53:46Z

Incompatibilities with existing code would definitely be a blocker. Those should not be possible, though. Tests for the original behavior still pass and remain unchanged.

Also, I'll need to do another rebase (and remove the redundant path >= 0 bit).

ChALkeR · 2015-10-08T11:20:29Z

fs.writeFile(-1), etc. will actually fail with TypeError: path must be a string, so there does not seem to be a type coercion backwards compatibility problem here.

Ah, I overlooked that. So then there should be no backwards-incompatibility here, and this is indeed a semver-minor only?

trevnorris · 2015-10-08T18:24:41Z

@ChALkeR This looks like an additive change. All previously accepted inputs are the same, with the addition of supporting uint32's for file descriptors. I'd say this is a semver-minor.

jwueller · 2015-10-13T18:09:31Z

@ChALkeR @trevnorris Rebased and updated!

trevnorris · 2015-10-13T18:22:56Z

lib/fs.js

-               0o666,
-               req);
+  if (context.isUserFd) {
+    req.oncomplete.call(req, null, path);


Put this in a nextTick. It's an expected async call, and must always run async.

jwueller · 2015-10-13T22:23:28Z

@trevnorris Updated again with the discussed changes.

trevnorris · 2015-10-13T22:43:35Z

doc/api/fs.markdown

+
+_Note: Specified file descriptors will not be closed automatically._
+
+## fs.appendFileSync(file, data[, options])


I see that you have the note in fs.writeFileSync, but not here in fs.appendFileSync. Is it necessary here?

I am not sure what you are asking, exactly. The note is present in all asynchronous versions, while being omitted in the synchronous ones, since the latter segments only refer to the asynchronous documentation anyway.

Whoop. Sorry. Got lost in the diff. Can see that now.

trevnorris · 2015-10-13T22:49:16Z

Almost there!

jwueller · 2015-10-13T23:09:53Z

After some further digging, I noticed that we have – in fact – several different approaches to dealing with file descriptor arguments available right now:

Method	`fd` Detection	`fd` Ownership Transfer
`fs.truncate()`	`typeof path === 'number'`	none
this PR	`(path >>> 0) === path`	none
`fs.createReadStream()` `fs.createWriteStream()`	supplied through `options.fd`, no type check	via `options.autoClose` (enabled by default)

This makes the PR in it's current form differ from existing functionality. It seems to me we should discuss which of these approaches is the right way to go here.

The latter one does not seem to have existed at the time of writing the original PR (and I did not want to make things more complicated than necessary at the time), but is definitely the most flexible option now that it exists "in the wild".

The behavior of fs.truncate() does not seem desirable. It is behaving strangely, as it automatically delegates to fs.ftruncate(), while other functions with direct fs.f*() counterparts do not. If this is legacy behavior, it likely could be officially deprecated to make the API more uniform. This should probably be the subject of a separate Issue/PR, though. Introducing commit is 168a555, for the record.

trevnorris · 2015-10-13T23:12:48Z

I believe how the functionality works in this PR is most appropriate of the differences. So I'd say we proceed with this then work on fixing the other implementations where necessary.

jwueller · 2015-10-13T23:47:22Z

That seems reasonable. Updated again.

trevnorris · 2015-10-14T16:26:06Z

Looks great. Running CI one last time for sanity then will land this thing: https://ci.nodejs.org/job/node-test-pull-request/508/

jwueller · 2015-10-14T20:56:52Z

Thank you very much for reviewing everything!

thefourtheye · 2015-10-15T02:20:04Z

lib/fs.js

  var req = new FSReqWrap();
  req.context = context;
  req.oncomplete = readFileAfterOpen;

+  if (context.isUserFd) {
+    process.nextTick(function() {
+      req.oncomplete.call(req, null, path);


Why not req.oncomplete(null, path)?

It would not behave correctly. This call is supposed to emulate the behavior of binding.open(). Omitting the context would lead to incompatibilities with following code (ex: readFileAfterOpen(), where the context is extracted from this).

the default context is the object to the left of the . so I think @thefourtheye is right in this case.

Of course he is. I actually tripped myself there. Sorry! I'll clean that up immediately.

jwueller · 2015-10-15T20:42:25Z

@trevnorris One more fix. Thanks @thefourtheye for reviewing!

These changes affect the following functions and their synchronous counterparts: * fs.readFile() * fs.writeFile() * fs.appendFile() If the first parameter is a uint32, it is treated as a file descriptor. In all other cases, the original implementation is used to ensure backwards compatibility. File descriptor ownership is never taken from the user. The documentation was adjusted to reflect these API changes. A note was added to make the user aware of file descriptor ownership and the conditions under which a file descriptor can be used by each of these functions. Tests were extended to test for file descriptor parameters under the conditions noted in the relevant documentation.

trevnorris · 2015-10-15T21:39:57Z

Final CI for sanity: https://ci.nodejs.org/job/node-test-pull-request/518/

If CI is good let's land this thing! LGTM

jwueller · 2015-10-16T22:30:11Z

@trevnorris Something seems to have failed, but it is not obvious to me what is happening, exactly. Can you help me out?

trevnorris · 2015-10-16T23:12:57Z

@jwueller None of the failing tests are related to your PR. Everything seems fine.

These changes affect the following functions and their synchronous counterparts: * fs.readFile() * fs.writeFile() * fs.appendFile() If the first parameter is a uint32, it is treated as a file descriptor. In all other cases, the original implementation is used to ensure backwards compatibility. File descriptor ownership is never taken from the user. The documentation was adjusted to reflect these API changes. A note was added to make the user aware of file descriptor ownership and the conditions under which a file descriptor can be used by each of these functions. Tests were extended to test for file descriptor parameters under the conditions noted in the relevant documentation. PR-URL: #3163 Reviewed-By: Trevor Norris <trev.norris@gmail.com>

trevnorris · 2015-10-16T23:16:09Z

Landed on 0803962. Thanks!

Note: This is labelled as semver-minor but it should not land on LTS.

jwueller · 2015-10-16T23:19:01Z

@trevnorris Great! Thanks again!

These changes affect the following functions and their synchronous counterparts: * fs.readFile() * fs.writeFile() * fs.appendFile() If the first parameter is a uint32, it is treated as a file descriptor. In all other cases, the original implementation is used to ensure backwards compatibility. File descriptor ownership is never taken from the user. The documentation was adjusted to reflect these API changes. A note was added to make the user aware of file descriptor ownership and the conditions under which a file descriptor can be used by each of these functions. Tests were extended to test for file descriptor parameters under the conditions noted in the relevant documentation. PR-URL: #3163 Reviewed-By: Trevor Norris <trev.norris@gmail.com>

Notable changes: * buffer: (Breaking) Removed both 'raw' and 'raws' encoding types from Buffer, these have been deprecated for a long time (Sakthipriyan Vairamani) nodejs#2859. * console: (Breaking) Values reported by console.time() now have 3 decimals of accuracy added (Michaël Zasso) nodejs#3166. * fs: - fs.readFile*(), fs.writeFile*(), and fs.appendFile*() now also accept a file descriptor as their first argument (Johannes Wüller) nodejs#3163. - (Breaking) In fs.readFile(), if an encoding is specified and the internal toString() fails the error is no longer thrown but is passed to the callback (Evan Lucas) nodejs#3485. - (Breaking) In fs.read() (using the fs.read(fd, length, position, encoding, callback) form), if the internal toString() fails the error is no longer thrown but is passed to the callback (Evan Lucas) nodejs#3503. * http: - Fixed a bug where pipelined http requests would stall (Fedor Indutny) nodejs#3342. - (Breaking) When parsing HTTP, don't add duplicates of the following headers: Retry-After, ETag, Last-Modified, Server, Age, Expires. This is in addition to the following headers which already block duplicates: Content-Type, Content-Length, User-Agent, Referer, Host, Authorization, Proxy-Authorization, If-Modified-Since, If-Unmodified-Since, From, Location, Max-Forwards (James M Snell) nodejs#3090. - (Breaking) The callback argument to OutgoingMessage#setTimeout() must be a function or a TypeError is thrown (James M Snell) nodejs#3090. - (Breaking) HTTP methods and header names must now conform to the RFC 2616 "token" rule, a list of allowed characters that excludes control characters and a number of separator characters. Specifically, methods and header names must now match /^[a-zA-Z0-9_!#$%&'*+.^`|~-]+$/ or a TypeError will be thrown (James M Snell) nodejs#2526. * node: - (Breaking) Deprecated the _linklist module (Rich Trott) nodejs#3078. - (Breaking) Removed require.paths and require.registerExtension(), both had been previously set to throw Error when accessed (Sakthipriyan Vairamani) nodejs#2922. * npm: Upgraded to version 3.3.6 from 2.14.7, see https://github.com/npm/npm/releases/tag/v3.3.6 for more details. This is a major version bump for npm and it has seen a significant amount of change. Please see the original npm v3.0.0 release notes for a list of major changes (Rebecca Turner) nodejs#3310. * src: (Breaking) Bumped NODE_MODULE_VERSION to 47 from 46, this is necessary due to the V8 upgrade. Native add-ons will need to be recompiled (Rod Vagg) nodejs#3400. * timers: Attempt to reuse the timer handle for setTimeout().unref(). This fixes a long-standing known issue where unrefed timers would perviously hold beforeExit open (Fedor Indutny) nodejs#3407. * tls: - Added ALPN Support (Shigeki Ohtsu) nodejs#2564. - TLS options can now be passed in an object to createSecurePair() (Коренберг Марк) nodejs#2441. - (Breaking) The default minimum DH key size for tls.connect() is now 1024 bits and a warning is shown when DH key size is less than 2048 bits. This a security consideration to prevent "logjam" attacks. A new minDHSize TLS option can be used to override the default. (Shigeki Ohtsu) nodejs#1831. * util: - (Breaking) util.p() was deprecated for years, and has now been removed (Wyatt Preul) nodejs#3432. - (Breaking) util.inherits() can now work with ES6 classes. This is considered a breaking change because of potential subtle side-effects caused by a change from directly reassigning the prototype of the constructor using `ctor.prototype = Object.create(superCtor.prototype, { constructor: { ... } })` to using `Object.setPrototypeOf(ctor.prototype, superCtor.prototype)` (Michaël Zasso) nodejs#3455. * v8: (Breaking) Upgraded to 4.6.85.25 from 4.5.103.35 (Ali Ijaz Sheikh) nodejs#3351. - Implements the spread operator, see https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Spread_operator for further information. - Implements new.target, see https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/new.target for further information. * zlib: Decompression now throws on truncated input (e.g. unexpected end of file) (Yuval Brik) nodejs#2595. PR-URL: nodejs#3466

Notable changes: * buffer: (Breaking) Removed both 'raw' and 'raws' encoding types from Buffer, these have been deprecated for a long time (Sakthipriyan Vairamani) #2859. * console: (Breaking) Values reported by console.time() now have 3 decimals of accuracy added (Michaël Zasso) #3166. * fs: - fs.readFile*(), fs.writeFile*(), and fs.appendFile*() now also accept a file descriptor as their first argument (Johannes Wüller) #3163. - (Breaking) In fs.readFile(), if an encoding is specified and the internal toString() fails the error is no longer thrown but is passed to the callback (Evan Lucas) #3485. - (Breaking) In fs.read() (using the fs.read(fd, length, position, encoding, callback) form), if the internal toString() fails the error is no longer thrown but is passed to the callback (Evan Lucas) #3503. * http: - Fixed a bug where pipelined http requests would stall (Fedor Indutny) #3342. - (Breaking) When parsing HTTP, don't add duplicates of the following headers: Retry-After, ETag, Last-Modified, Server, Age, Expires. This is in addition to the following headers which already block duplicates: Content-Type, Content-Length, User-Agent, Referer, Host, Authorization, Proxy-Authorization, If-Modified-Since, If-Unmodified-Since, From, Location, Max-Forwards (James M Snell) #3090. - (Breaking) The callback argument to OutgoingMessage#setTimeout() must be a function or a TypeError is thrown (James M Snell) #3090. - (Breaking) HTTP methods and header names must now conform to the RFC 2616 "token" rule, a list of allowed characters that excludes control characters and a number of separator characters. Specifically, methods and header names must now match /^[a-zA-Z0-9_!#$%&'*+.^`|~-]+$/ or a TypeError will be thrown (James M Snell) #2526. * node: - (Breaking) Deprecated the _linklist module (Rich Trott) #3078. - (Breaking) Removed require.paths and require.registerExtension(), both had been previously set to throw Error when accessed (Sakthipriyan Vairamani) #2922. * npm: Upgraded to version 3.3.6 from 2.14.7, see https://github.com/npm/npm/releases/tag/v3.3.6 for more details. This is a major version bump for npm and it has seen a significant amount of change. Please see the original npm v3.0.0 release notes for a list of major changes (Rebecca Turner) #3310. * src: (Breaking) Bumped NODE_MODULE_VERSION to 47 from 46, this is necessary due to the V8 upgrade. Native add-ons will need to be recompiled (Rod Vagg) #3400. * timers: Attempt to reuse the timer handle for setTimeout().unref(). This fixes a long-standing known issue where unrefed timers would perviously hold beforeExit open (Fedor Indutny) #3407. * tls: - Added ALPN Support (Shigeki Ohtsu) #2564. - TLS options can now be passed in an object to createSecurePair() (Коренберг Марк) #2441. - (Breaking) The default minimum DH key size for tls.connect() is now 1024 bits and a warning is shown when DH key size is less than 2048 bits. This a security consideration to prevent "logjam" attacks. A new minDHSize TLS option can be used to override the default. (Shigeki Ohtsu) #1831. * util: - (Breaking) util.p() was deprecated for years, and has now been removed (Wyatt Preul) #3432. - (Breaking) util.inherits() can now work with ES6 classes. This is considered a breaking change because of potential subtle side-effects caused by a change from directly reassigning the prototype of the constructor using `ctor.prototype = Object.create(superCtor.prototype, { constructor: { ... } })` to using `Object.setPrototypeOf(ctor.prototype, superCtor.prototype)` (Michaël Zasso) #3455. * v8: (Breaking) Upgraded to 4.6.85.25 from 4.5.103.35 (Ali Ijaz Sheikh) #3351. - Implements the spread operator, see https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Spread_operator for further information. - Implements new.target, see https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/new.target for further information. * zlib: Decompression now throws on truncated input (e.g. unexpected end of file) (Yuval Brik) #2595. PR-URL: #3466

mscdex added the fs Issues and PRs related to the fs subsystem / file system. label Oct 3, 2015

ChALkeR added the semver-minor PRs that contain new features and should be released in the next minor version. label Oct 4, 2015

ChALkeR reviewed Oct 4, 2015
View reviewed changes

trevnorris reviewed Oct 13, 2015
View reviewed changes

thefourtheye reviewed Oct 15, 2015
View reviewed changes

trevnorris closed this Oct 16, 2015

jwueller deleted the fs-optional-file-descriptor-args branch October 17, 2015 00:09

rvagg mentioned this pull request Oct 21, 2015

Release proposal: v5.0.0 #3466

Merged

thisconnect mentioned this pull request Dec 15, 2015

add tests for passing fd to fs.file* only Node 5.x thisconnect/fildes#17

Closed

targos mentioned this pull request Mar 23, 2016

fs.readFile don't work with file descriptor #5862

Closed

MylesBorins added the dont-land-on-v4.x label Jan 6, 2017

sam-github mentioned this pull request Jan 17, 2017

doc: clarify the behavior of readFile with fd #10853

Closed

2 tasks

thisconnect mentioned this pull request Jun 9, 2017

'fs' can not read data if same FD is used for write and readFile #13572

Closed


		_Note: Specified file descriptors will not be closed automatically._

		## fs.appendFileSync(file, data[, options])

fs: Add file descriptor support to *File() funcs #3163

fs: Add file descriptor support to *File() funcs #3163

Conversation

jwueller commented Oct 3, 2015

jwueller commented Oct 3, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ChALkeR commented Oct 4, 2015

jwueller commented Oct 4, 2015

trevnorris commented Oct 7, 2015

jwueller commented Oct 8, 2015

ChALkeR commented Oct 8, 2015

trevnorris commented Oct 8, 2015

jwueller commented Oct 13, 2015

Choose a reason for hiding this comment

jwueller commented Oct 13, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

trevnorris commented Oct 13, 2015

jwueller commented Oct 13, 2015

trevnorris commented Oct 13, 2015

jwueller commented Oct 13, 2015

trevnorris commented Oct 14, 2015

jwueller commented Oct 14, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jwueller commented Oct 15, 2015

trevnorris commented Oct 15, 2015

jwueller commented Oct 16, 2015

trevnorris commented Oct 16, 2015

trevnorris commented Oct 16, 2015

jwueller commented Oct 16, 2015