Skip to content
This repository has been archived by the owner on Jan 20, 2022. It is now read-only.

Make dependencies' sort locale independent #276

Closed
wants to merge 1 commit into from
Closed

Make dependencies' sort locale independent #276

wants to merge 1 commit into from

Conversation

padinko
Copy link

@padinko padinko commented May 5, 2021

this will fix npm's sorting of dependencies in package.json npm/cli#2829

now dependencies in package.json are sorted locale dependent, which makes npm unusable on non-english systems

same problem was fixed in 2017 https://github.com/npm/npm/pull/17844/files but now sorting package.json and package-lock.json are again locale independend in npm 7

for me this is a big issue, because some of us in our company is working with english OS, some in slovak and both package.json and package-lock files are changing with every npm install

if you are on *nix system, you can test LC_ALL=sk npm i and LC_ALL=en-US npm i (as described in issue above). but on windows there is no way to change sort language.

npm 6 from 2017 fix use native .sort(), which use C.UTF-8 sort, now npm 7 use localeCompare, which changed sorting even on English systems: - is now after _.
so fixing this by forcing sorting to en-US locale makes npm 6 and npm 7 dependencies sorted different

@wraithgar
Copy link
Member

wraithgar commented May 5, 2021

At issue here isn't the differences between 6 and 7, its the idempotency of the action. The results shouldn't differ based on end user locale.

This PR is going to be unable to merge after #273 is landed, because this function was inlined into update-root-package-json.js.

So while I think this is a good change, we'll have to wait for the other PR to land so this one can be re-applied there. We will also want to have a test that goes inside the deps are alphabetized block in test/update-root-package-json.js that shows this in action.

@isaacs
Copy link
Contributor

isaacs commented May 5, 2021

Other places where we use localeCompare, probably worth moving wholesale to a locale-agnostic sorting throughout npm/cli and its dependencies:

lib/config.js
lib/help.js
lib/ls.js
lib/outdated.js
lib/utils/completion/installed-deep.js
lib/utils/config/describe-all.js
lib/utils/npm-usage.js
lib/utils/tar.js
node_modules/@npmcli/arborist/bin/license.js
node_modules/@npmcli/arborist/lib/add-rm-pkg-deps.js
node_modules/@npmcli/arborist/lib/arborist/build-ideal-tree.js
node_modules/@npmcli/arborist/lib/arborist/load-virtual.js
node_modules/@npmcli/arborist/lib/arborist/rebuild.js
node_modules/@npmcli/arborist/lib/audit-report.js
node_modules/@npmcli/arborist/lib/dep-spec.js
node_modules/@npmcli/arborist/lib/printable.js
node_modules/@npmcli/arborist/lib/shrinkwrap.js
node_modules/@npmcli/arborist/lib/vuln.js
node_modules/@npmcli/arborist/lib/yarn-lock.js
node_modules/colors/lib/extendStringPrototype.js
node_modules/glob/common.js
node_modules/ignore-walk/index.js
node_modules/json-stringify-nice/index.js
node_modules/libnpmexec/lib/cache-install-dir.js
node_modules/libnpmexec/lib/index.js
node_modules/npm-packlist/bin/index.js
node_modules/npm-packlist/index.js
scripts/bundle-and-gitignore-deps.js
scripts/config-doc.js
test/lib/exec.js
test/lib/link.js
test/lib/load-all-commands.js
test/lib/utils/cleanup-log-files.js

@isaacs
Copy link
Contributor

isaacs commented May 6, 2021

Ok, new plan.

Instead of dropping localeCompare for ascii sort() order, we're going to specify 'en' as the second argument to localeCompare everywhere. Internationalization support was added in node 0.12, but was optional until 13, and if not built with full intl support, only the 'en' locale is used, regardless of the argument.

isaacs added a commit that referenced this pull request May 6, 2021
Node will use the current environment's locale sorting for
String.localeCompare.  Thankfully, String.localeCompare can take a
second argument to specify the locale.  Up until Node 13, any valid
argument provided to the function _may_ be respected if Node was
compiled with Intl support, or would always be treated as `'en'`
otherwise.

The solution is to always set the locale to `'en'`.  An alternative
solution would be to compare strings based on byte order, but this does
not provide ideal ergonomis.

To verify this, the locale is set in the test environment to `'sk'`,
which has a significantly different alphabet, including sorting words
starting with `'ch'` _after_ words starting with `'d'`.

Re: npm/cli#2829
Fix: #276
isaacs added a commit that referenced this pull request May 6, 2021
Node will use the current environment's locale sorting for
String.localeCompare.  Thankfully, String.localeCompare can take a
second argument to specify the locale.  Up until Node 13, any valid
argument provided to the function _may_ be respected if Node was
compiled with Intl support, or would always be treated as `'en'`
otherwise.

The solution is to always set the locale to `'en'`.  An alternative
solution would be to compare strings based on byte order, but this does
not provide ideal ergonomis.

To verify this, the locale is set in the test environment to `'sk'`,
which has a significantly different alphabet, including sorting words
starting with `'ch'` _after_ words starting with `'d'`.

Re: npm/cli#2829
Fix: #276
isaacs added a commit that referenced this pull request May 7, 2021
Node will use the current environment's locale sorting for
String.localeCompare.  Thankfully, String.localeCompare can take a
second argument to specify the locale.  Up until Node 13, any valid
argument provided to the function _may_ be respected if Node was
compiled with Intl support, or would always be treated as `'en'`
otherwise.

The solution is to always set the locale to `'en'`.  An alternative
solution would be to compare strings based on byte order, but this does
not provide ideal ergonomis.

To verify this, the locale is set in the test environment to `'sk'`,
which has a significantly different alphabet, including sorting words
starting with `'ch'` _after_ words starting with `'d'`.

Re: npm/cli#2829
Fix: #276
@isaacs
Copy link
Contributor

isaacs commented May 8, 2021

Going with the approach in #280. Thanks for bringing this to our attention!

@isaacs isaacs closed this May 8, 2021
isaacs added a commit that referenced this pull request May 10, 2021
Node will use the current environment's locale sorting for
String.localeCompare.  Thankfully, String.localeCompare can take a
second argument to specify the locale.  Up until Node 13, any valid
argument provided to the function _may_ be respected if Node was
compiled with Intl support, or would always be treated as `'en'`
otherwise.

The solution is to always set the locale to `'en'`.  An alternative
solution would be to compare strings based on byte order, but this does
not provide ideal ergonomis.

To verify this, the locale is set in the test environment to `'sk'`,
which has a significantly different alphabet, including sorting words
starting with `'ch'` _after_ words starting with `'d'`.

Re: npm/cli#2829
Fix: #276
isaacs added a commit that referenced this pull request May 10, 2021
Node will use the current environment's locale sorting for
String.localeCompare.  Thankfully, String.localeCompare can take a
second argument to specify the locale.  Up until Node 13, any valid
argument provided to the function _may_ be respected if Node was
compiled with Intl support, or would always be treated as `'en'`
otherwise.

The solution is to always set the locale to `'en'`.  An alternative
solution would be to compare strings based on byte order, but this does
not provide ideal ergonomis.

To verify this, the locale is set in the test environment to `'sk'`,
which has a significantly different alphabet, including sorting words
starting with `'ch'` _after_ words starting with `'d'`.

Re: npm/cli#2829
Fix: #276
wraithgar pushed a commit that referenced this pull request May 10, 2021
Node will use the current environment's locale sorting for
String.localeCompare.  Thankfully, String.localeCompare can take a
second argument to specify the locale.  Up until Node 13, any valid
argument provided to the function _may_ be respected if Node was
compiled with Intl support, or would always be treated as `'en'`
otherwise.

The solution is to always set the locale to `'en'`.  An alternative
solution would be to compare strings based on byte order, but this does
not provide ideal ergonomis.

To verify this, the locale is set in the test environment to `'sk'`,
which has a significantly different alphabet, including sorting words
starting with `'ch'` _after_ words starting with `'d'`.

Re: npm/cli#2829
Fix: #276
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants