src,fs: calculate times as unsigned long #13281

refack · 2017-05-29T12:36:24Z

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
commit message follows commit guidelines

Affected core subsystem(s)

fs

gibfahn · 2017-05-29T16:02:10Z

test/parallel/test-fs-utimes.js

  assert.strictEqual(tests_ok, tests_run);
-});
+});


Needs a newline at the end of the file

refack · 2017-05-29T16:04:46Z

"Dry run" CI: https://ci.nodejs.org/job/node-test-commit/10219/

addaleax · 2017-05-29T16:08:46Z

test/parallel/test-fs-utimes.js

@@ -164,7 +164,23 @@ runTest(new Date('1982-09-10 13:37'), new Date('1982-09-10 13:37'), function() {
 });


+if (common.isWindows) {
+  common.refreshTempDir


This should be a call, right?

addaleax · 2017-05-29T16:12:59Z

test/parallel/test-fs-utimes.js

+  const path = `${common.tmpDir}/test-utimes-precision`;
+  fs.writeFileSync(path, '');
+
+  const truncate_mtime = 1713037251360;


I think it would help with readability if you could add a comment explaining why these magic values are magic (I had to dig into the libuv sources, and I’m still not quite sure I got it right). ;)

Tryied to hint with the name, but will add a comment.

addaleax · 2017-05-29T16:18:05Z

src/node_file.cc

-                (s->st_##name.tv_nsec / 1e6); \
+#define X(idx, name)                                                      \
+  fields[idx] = (static_cast<unsigned long>(s->st_##name.tv_sec) * 1e3) + \
+                (static_cast<unsigned long>(s->st_##name.tv_nsec) / 1e6); \


I think it would be a good idea if you could leave a comment here, explaining why this is needed.

And just so I get it right; this is needed because tv_sec overflows? I think the real problem in that case is that we don’t check for overflows in libuv during fs__utime_handle, which we probably should? Am I getting it totally wrong?

refack · 2017-05-29T16:55:38Z

delint @ added comments
new "Dry Run" CI: https://ci.nodejs.org/job/node-test-commit/10223/

addaleax · 2017-05-29T16:57:01Z

src/node_file.cc

-  fields[3] = s->st_uid;
-  fields[4] = s->st_gid;
-  fields[5] = s->st_rdev;
+  fields[0] = static_cast<double>(s->st_dev);


These changes seem unrelated, and I don’t think they are actually changing anything?

They remove warnings, but I'll kick them out since I have a more "warning reducing" PR cooking

addaleax · 2017-05-29T16:58:45Z

test/parallel/test-fs-utimes.js

+  const truncate_stats = fs.statSync(path);
+  assert.strictEqual(truncate_mtime, truncate_stats.mtime.getTime());
+
+  // if this value is cast to `signed long` it gets converted to -2135622133469


That’s not really a comment explaining what’s happening…

It’s not like (signed long) 2159345162531 == -2135622133469 – it’s more complicated than that, it definitely relates to how the Windows API deals with timestamps.

It's almost that simple https://github.com/nodejs/node/blob/master/deps/uv/src/win/fs.c#L91
NTFS stores times in nanoseconds in a uint64_t but when uv converts it down to long we get 2's complement.

Hm. Then maybe we should cast back to unsigned only on Windows?

But really, I’m not sure whether this is the right approach, or if we are better off throwing in utimes if the time is not representable as a long…

I added to the comment why it's not a problem in POSIX, because there timestamps are stored in two fields struct {long t_sec, long t_usec}, and neither can overflow.
The problem is just how we convert from the NTFS uint64_t that hold nanoseconds, back to seconds.
IMHO This change is safe, since even the numbers from NTFS if properly converted to seconds fit in a long and in a double. (notice that the actual call to utimesSync divides these numbers by 1000). Also no FS will give us negative times, and the paper mentioned in the bug (interpretation-of-ntfs-timestamps), says we just need to interpret it right.

You might be right in that the calculations in https://github.com/nodejs/node/blob/master/deps/uv/src/win/fs.c#L91 should be done with uint

refack · 2017-05-29T17:50:30Z

more cleanup.
new "Dry Run" CI: https://ci.nodejs.org/job/node-test-commit/10226/

addaleax

Thanks for the updates! Looks good assuming we’re switching back to not doing 32-bit math.

addaleax · 2017-05-29T18:10:07Z

src/node_file.cc

-                (s->st_##name.tv_nsec / 1e6); \
+#define X(idx, name)                                                 \
+  fields[idx] = (static_cast<uint32_t>(s->st_##name.tv_sec) * 1e3) + \
+                (static_cast<uint32_t>(s->st_##name.tv_nsec) / 1e6); \


Y2k38 bug? ;) I think doing the math in uint64_t (or double because it’s getting converted anyway) as you had it before was fine, is there any reason to switch?

This whole fix turns Y2K38 to something like Y3K16
But there's a bug when doing static_cast<uint64_t> since the MSB is 1, it blows up to a huge number.
Also long (a.k.a. int32_t) to uint32_t is faster / stabler anyway.

*static_cast<uint64_t>

This whole fix turns Y2K38 to something like Y3K16

Ah, right. (Still not great, but yes, it’s better.)

But there's a bug when doing static_cast<uint64_t> since the MSB is 1, it blows up to a huge number.

You have the overflow problem anyway, though? Also, this doesn’t happen if you cast to double directly.

since casting to double is a calculation is comes out a negative double.
Maybe a reinterpret_cast<uint64_t> or static_cast<uint64_t>(static_cast<uint32_t>()) first inner one to convince the compiler it's a positive number...

@refack The thing is, I’m not sure you can count on long being 32-bit on all those odd systems that Node supports. unsigned long is the right thing anyway if you are reading from a long.

That was my initial thought but the linter was angry at me:
for
(static_cast<unsigned long>(s->st_##name.tv_sec) * 1e3)
it spet out
src/node_file.cc:483: Use int16/int64/etc, rather than the C type long [runtime/int] [4]
got the same for:
(unsigned long)(s->st_##name.tv_sec) * 1e3)

That was my initial thought

Good that we agree, though. :)

but the linter was angry at me:

// NOLINT(runtime/int) should help, although I’m not sure where exactly you need to place that

addaleax · 2017-05-29T18:11:57Z

test/parallel/test-fs-utimes.js

+
+  // This value if treaded as a `signed long` gets converted to -2135622133469.
+  // POSIX systems stores timestamps in {long t_sec, long t_usec}.
+  // NTFS stores times in nanoseconds in a single `uint64_t`, so when `uv`


s/uv/libuv/

addaleax

Thank you!

refack · 2017-05-29T19:49:14Z

Thank you!

No thank you!

Just linter: https://ci.nodejs.org/job/node-test-linter/9444/
windows: https://ci.nodejs.org/job/node-test-commit-windows-fanned/9436/
and linuxone: https://ci.nodejs.org/job/node-test-commit-linuxone/6238/

refack · 2017-06-01T01:59:42Z

@sciolist This looks like the other half of #12607 now we're fixing Y2k38

sciolist · 2017-06-01T09:51:54Z

fs dates on windows is a gift that keeps on giving.

Would it be bad to have the 2k38 test use 2159345163000 and let it run on non-windows as well?

refack · 2017-06-01T14:19:11Z

Would it be bad to have the 2k38 test use 2159345163000 and let it run on non-windows as well?

good idea 👍 (even though there it's kept as seconds, not millis)

refack · 2017-06-01T14:58:09Z

"Dry run" CI: https://ci.nodejs.org/job/node-test-commit/10275/

sciolist · 2017-06-01T17:07:48Z

@refack yeah that's why i suggested increasing the number to a full second, but your method is nicer and a lot clearer what it's testing for. I was interested in seeing if all the other platforms would handle 2k38, seems like most of them did fine at least. :)

refack · 2017-06-01T18:53:13Z

Dry run CI:https://ci.nodejs.org/job/node-test-commit/10283/

refack · 2017-06-27T16:37:25Z

Rerun CI: https://ci.nodejs.org/job/node-test-commit/10787/

refack · 2017-06-27T21:02:15Z

Preland extra sanity: ~~https://ci.nodejs.org/job/node-test-commit-linuxone/6898/~~
~~https://ci.nodejs.org/job/node-test-commit-linuxone/6899/~~
https://ci.nodejs.org/job/node-test-commit-linuxone/6900/

PR-URL: nodejs#13281 Fixes: nodejs#13255 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: James M Snell <jasnell@gmail.com>

PR-URL: #13281 Fixes: #13255 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: James M Snell <jasnell@gmail.com>

MylesBorins · 2017-08-14T20:35:49Z

Should this land in LTS?

Does it have any chance of changing behavior?

refack · 2017-08-14T22:45:49Z

Does it have any chance of changing behavior?

~~Not~~ No change in correct behaviour. It solves Y2K38 on ~~linux~~ POSIX, and enables higher precision times on Windows for use by #13173 (noop on current Stats otherwise)

MylesBorins · 2017-09-19T15:52:59Z

This is not landing cleanly, ca you please backport with relevant commits

nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. fs Issues and PRs related to the fs subsystem / file system. labels May 29, 2017

gibfahn reviewed May 29, 2017

View reviewed changes

refack force-pushed the more-for-12607 branch from 746b413 to 977e103 Compare May 29, 2017 16:03

refack force-pushed the more-for-12607 branch from 977e103 to 510ed52 Compare May 29, 2017 16:13

addaleax reviewed May 29, 2017

View reviewed changes

refack mentioned this pull request May 29, 2017

utimes precision errors #13255

Closed

refack force-pushed the more-for-12607 branch from 510ed52 to 21d8770 Compare May 29, 2017 16:53

addaleax reviewed May 29, 2017

View reviewed changes

refack force-pushed the more-for-12607 branch 2 times, most recently from 297a69c to 1e4c0c8 Compare May 29, 2017 17:49

addaleax approved these changes May 29, 2017

View reviewed changes

refack force-pushed the more-for-12607 branch from beb9ad4 to 342c2e2 Compare June 1, 2017 14:49

refack force-pushed the more-for-12607 branch from ccd152f to 7c5b0da Compare June 1, 2017 18:49

jasnell approved these changes Jun 1, 2017

View reviewed changes

refack force-pushed the more-for-12607 branch from 7c5b0da to e36be12 Compare June 27, 2017 21:01

src,fs: calculate times as unsigned long

0526f84

PR-URL: nodejs#13281 Fixes: nodejs#13255 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: James M Snell <jasnell@gmail.com>

refack force-pushed the more-for-12607 branch from e36be12 to 0526f84 Compare June 27, 2017 21:11

refack merged commit 0526f84 into nodejs:master Jun 27, 2017

addaleax pushed a commit that referenced this pull request Jun 29, 2017

src,fs: calculate times as unsigned long

56e282b

PR-URL: #13281 Fixes: #13255 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: James M Snell <jasnell@gmail.com>

addaleax mentioned this pull request Jun 29, 2017

v8.2.0 proposal #13744

Merged

refack deleted the more-for-12607 branch July 2, 2017 02:34

bnoordhuis mentioned this pull request Jul 10, 2017

fs.utimes truncate date #13520

Closed

addaleax pushed a commit that referenced this pull request Jul 11, 2017

src,fs: calculate times as unsigned long

a1e2423

PR-URL: #13281 Fixes: #13255 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: James M Snell <jasnell@gmail.com>

refack mentioned this pull request Jul 17, 2017

src: round nsec instead of truncating stat times #12607

Closed

3 tasks

addaleax pushed a commit that referenced this pull request Jul 18, 2017

src,fs: calculate times as unsigned long

5579bc8

PR-URL: #13281 Fixes: #13255 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: James M Snell <jasnell@gmail.com>

MylesBorins added the backport-requested-v6.x label Sep 19, 2017

sciolist mentioned this pull request Sep 20, 2017

fs: expose Stats times as Numbers #13173

Merged

4 tasks

bnoordhuis mentioned this pull request Dec 29, 2017

parallel/test-fs-utimes y2k38 test fails on mips(el) #17909

Closed

src,fs: calculate times as unsigned long #13281

src,fs: calculate times as unsigned long #13281

Conversation

refack commented May 29, 2017

Checklist

Affected core subsystem(s)

Choose a reason for hiding this comment

Choose a reason for hiding this comment

refack commented May 29, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

refack commented May 29, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

refack commented May 29, 2017

addaleax left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

refack May 29, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

refack May 29, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

addaleax left a comment

Choose a reason for hiding this comment

refack commented May 29, 2017

refack commented Jun 1, 2017

sciolist commented Jun 1, 2017

refack commented Jun 1, 2017

refack commented Jun 1, 2017

sciolist commented Jun 1, 2017

refack commented Jun 1, 2017

refack commented Jun 27, 2017

refack commented Jun 27, 2017 • edited Loading

MylesBorins commented Aug 14, 2017

refack commented Aug 14, 2017 • edited Loading

MylesBorins commented Sep 19, 2017

refack May 29, 2017 •

edited

Loading

refack May 29, 2017 •

edited

Loading

refack commented Jun 27, 2017 •

edited

Loading

refack commented Aug 14, 2017 •

edited

Loading