(Windows) FS can not handle certain characters in file name

[ZEDCWT]

Version

v20.4.0

Platform

Microsoft Windows NT 10.0.19045.0 x64

Subsystem

No response

What steps will reproduce the bug?

Run the following script

require('fs').writeFileSync('\uD83D\uDD79\uFE0F.log','Test')

Where \uD83D\uDD79\uFE0F is 🕹️

How often does it reproduce? Is there a required condition?

Always failed for v20.4.0, v20.3.1 is okay

What is the expected behavior? Why is that the expected behavior?

Run without errors

What do you see instead?

Additional information

Tried on linux, no errors

[bnoordhuis]

Libuv recently switched to WTF-8, to deal with surrogate pairs (0xD83D 0xDD79 is a surrogate pair) in Windows file paths. That change landed in v20.4.0.

cc @vtjnash

[vtjnash]

\uD83D\uDD79\uFE0F is not a legal string, since it incorrectly encodes for a surrogate, not a unicode character. The Joystick emoji encoding is:

U+1F579 ️ U+FE0F

[vtjnash]

We could potentially disable the UTF-8 transcoder error checking again though, and permit this to access the file anyways. The name would not round-trip correctly through the file-system, but that is not too significantly different from how case-sensitivity can be lost during the conversion to accessing the file system.

[ZEDCWT]

\uD83D\uDD79\uFE0F is not a legal string, since it incorrectly encodes for a surrogate, not a unicode character. The Joystick emoji encoding is:

U+1F579 ️ U+FE0F

For what reason you think it is not valid?

'\u{1F579}'==='\uD83D\uDD79' // true

This is JavaScript, they are identical.

[vtjnash]

Ah, interesting. That means node is explicitly converting the valid string in ecmascript's UCS-2 into an invalid UTF-8 string, possibly intentionally since ecmascript does not actually support this emoji and instead defines that it is composed of 2 unknown characters from the surrogate block. We could certainly change libuv to be a non-validating UTF-8 decoder so that it can still accept invalid strings as input.

https://mathiasbynens.be/notes/javascript-encoding

[rvagg]

This maybe shouldn't be closed until we get a libuv release then a Node release that includes it? We're going to collect a lot of dupes for this and they should be closed and point here until the release is done. #48813 #49042.

(Speaking from experience, I'm dealing with some failing windows CI tests since 20.4.0 https://github.com/ipld/codec-fixtures/actions/runs/5756528910/job/15606085523 for files that have ... complicated ... names).

[user7230724]

LTS is also affected. Works in v18.0.0, but not in v18.18.0.

[bnoordhuis]

It's fixed upstream in libuv/libuv@d09441c but not released yet.

[qianz2]

We're using 20.3.1 version, could we had the fix backported into this version as well?

[AlttiRi]

It was finally fixed in LTS(5405aa5).

• [v20.x backport] deps: upgrade libuv to 1.48.0 #51699

---

WTF?

import fs from "node:fs/promises";
await fs.writeFile("🚀🔥🛸.txt", "");

node:internal/fs/promises:609
    await binding.openFileHandle(pathModule.toNamespacedPath(path),
                  ^

Error: ENOENT: no such file or directory, open
    at open (node:internal/fs/promises:609:19)
    at Object.writeFile (node:internal/fs/promises:1055:20)
    at file:///C:/Dev/test/x.mjs:2:10
    at ModuleJob.run (node:internal/modules/esm/module_job:218:25)
    at async ModuleLoader.import (node:internal/modules/esm/loader:329:24)
    at async loadESM (node:internal/process/esm_loader:28:7)
    at async handleMainPromise (node:internal/modules/run_main:113:12) {
  errno: -4058,
  code: 'ENOENT',
  syscall: 'open'
}

Node.js v20.11.1

Also, I've seen these errors:

node:internal/fs/utils:557
    stats[0 + offset], stats[1 + offset], stats[2 + offset],
         ^

TypeError: Cannot read properties of undefined (reading '0')
    at getStatsFromBinding (node:internal/fs/utils:557:10)
    at Object.lstat (node:internal/fs/promises:917:10)
    at async file:///C:/Dev/test/x.mjs:3:1

node:internal/fs/promises:915
  const result = await binding.lstat(pathModule.toNamespacedPath(path),
                               ^

Error: ENOENT: no such file or directory, lstat
    at Object.lstat (node:internal/fs/promises:915:32)
    at file:///C:/Downloads/1/x.mjs:3:16
    at ModuleJob.run (node:internal/modules/esm/module_job:218:25)
    at async ModuleLoader.import (node:internal/modules/esm/loader:329:24)
    at async loadESM (node:internal/process/esm_loader:28:7)
    at async handleMainPromise (node:internal/modules/run_main:113:12) {
  errno: -4058,
  code: 'ENOENT',
  syscall: 'lstat'
}

Node.js v20.11.1

It's still not fixed. Even with libuv 1.48.

Even Bun https://github.com/oven-sh/bun/ works fine with it.

---

UPD.

However, it's fixed in Current (v21.6.2).

But it's not fixed in LTS (v20.11.1) that was released 3 day ago.

---

> LTS (Long-Term Support). Only verified changes.
> "Recommended For Most Users"
> broken

Are you kidding?

---

Node.js even have not tests for reading/writing files with surrogate pairs in a filename?

[Rinse12]

Do we have a release date for this bug fix?

[AlttiRi]

I assume, in the next LTS release after merging that PR:

• [v20.x] backport libuv wtf-8 decoding fix #51976

---

It takes more than half a year to add 3 lines of code.

[AlttiRi]

Finally, the fix was backported to LTS (v20.12.0).

• v20.12.0 proposal #52212:

  • [cb49f31480] - deps: cherry-pick libuv/libuv@d09441c (Richard Lau) [v20.x] backport libuv wtf-8 decoding fix #51976
  • [e8f5735149] - test: test surrogate pair filenames on windows (Mert Can Altın) test: Test surrogate pair filenames on windows #51800

  Rel: Add tests for filenames with surrogate pairs on Windows #51789