Bundling/preinstall woes

[springmeyer]

Ticket to keep track of edge cases around applications bundling node-pre-gyp (which was the original recommendation but alternatives were proposed in #162) and not bundling (which was documented as an option in #201).

• Module uses node-pre-gyp, does bundle node-pre-gyp, and has a direct dependency on a dependency of node-pre-gyp - likely will be de-duped and breaks node v0.10.x install when node-pre-gyp is bundled and the dep is missing: v0.5.10 broken for node 0.10.x node-zipfile#72
• Module uses node-pre-gyp and does not bundle node-pre-gyp: likely will break windows: 3.1.5 breaks global install of dependents on Windows TryGhost/node-sqlite3#720
• Module uses node-pre-gyp and does not bundle node-pre-gyp and uses npm install node-pre-gyp in "preinstall": may lead to unexplainable hangs with various npm versions: Infinite loop during preisntall on node 0.12.7 npm 3.8.8 npm/npm#12583 | rebundle node-pre-gyp and do not preinstall it mapnik/node-mapnik#658
• Module uses node-pre-gyp and does not bundle node-pre-gyp and uses npm install node-pre-gyp in "preinstall": preinstall fails with enoent ENOENT: no such file or directory, open '/home/user/<package>/package.json' - refs Installation issues with node v4.2.6, npm v3.5.2 Project-OSRM/node-osrm#305

More details: fsevents/fsevents#157 (comment)

[tjwebb]

may lead to unexplainable hangs with various npm versions:

They are not unexplainable, they are a logical result of using npm/package.json in a way they are not designed to be used. You've already read my explanation: mapnik/node-mapnik#650 (comment)

[reconbot]

So as of npm3 I think we still have to bundle, I haven't checked in a while. Also to support npm2 installs I've found I have to publish (and reinstall node-modules) with npm2. With bundling and npm2 publishing I've been able to have issue free installs on all platforms and node versions (including 10 and 12) and I've gone as far as to test it. https://github.com/johnny-five-io/serialport-test-pilot I still think npm/npm#12583 is a bug however all that means is we can't do that. Even if it's fixed we still should package for the versions of npm that ship with the versions of node we support. ¯\_(ツ)_/¯

…

On Sat, Jan 21, 2017, 10:04 PM Travis Webb ***@***.***> wrote: may lead to unexplainable hangs with various npm versions: They are not unexplainable, they are a logical result of intentionally using npm/package.json in a way they are not designed to be used. You've already read my explanation: mapnik/node-mapnik#650 (comment) <mapnik/node-mapnik#650 (comment)> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#260 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABlbklPUCnPP321qSm1f-yIHzx_2ptGks5rUsdJgaJpZM4LOy6F> .

[tjwebb]

Hey @reconbot! Thanks for visiting us in Norfolk yesterday.

and has a direct dependency on a dependency of node-pre-gyp - likely will be de-duped and breaks node v0.10.x install when node-pre-gyp is bundled and the dep is missing

This can be solved by setting NODE_PATH=node_modules. Running a separate npm install inside of npm install feels like a really heavy-handed solution for a path problem. This creates issues newer (npm3+) node versions in order to fix issues in older (~npm2) node versions. The direction of this tradeoff seems backwards to me. If breakage is required, we should break old versions, not new ones.

Another issue here is some modules' reliance on the particular way npm2/npm3 structure the node_modules folder. Some of these issues can be avoided by being a little bit cute with the paths in binding.gyp, e.g. https://github.com/langateam/node-mapnik/blob/master/binding.gyp#L60-L61. This bullet point describes a slightly different issue, but it's in the same theme of path problems introduced by different npm versions.

[springmeyer]

Thanks everyone for the feedback. I've reverted #201 by applying #269, which changes the documentation to recommend bundling. Because this is only a documentation change the work ahead then is to:

• Find modules using the preinstall method
• Add back bundledDependencies and the "prepublish": "npm ls" (the latter to help avoid incorrect bundling)
• Use this ticket to drop ideas or problems with ^^

/cc @tjwebb @reconbot

[SomeoneElseOSM]

What's the best way to handle "anecdotes of failure"? I've just had the familiar

npm ERR! Tell the author that this fails on your system:
npm ERR!     node-pre-gyp install --fallback-to-build

on another system but I presume that you don't want this issue clogged up with with each individual failure and stacktraces. Are there any particular bits of information that are useful to obtain?

(for completeness, in case it invalidates this as relevant to this issue - this was with the LTS version of node as recommended at kosmtik/kosmtik#232 (comment) ).

[springmeyer]

@SomeoneElseOSM kosmtik/kosmtik#232 is likely resolvable by downgrading to npm to 2.x or upgrading node-mapnik to 3.6.x.

[pronebird]

I always get X is missing a bundled dependency "node-pre-gyp". Is it ok for modules shipped over GitHub and not npm?

[springmeyer]

I always get X is missing a bundled dependency "node-pre-gyp". Is it ok for modules shipped over GitHub and not npm?

This is expected/normal @pronebird - generally this is okay for modules coming via github, because (if I recall correctly), in this case npm will not dedupe. This lack of deduping avoids one of the problems that requires bundling in the case of an npm install: which is that:

• you might have a project that depends on > 1 module that uses node-pre-gyp
• npm might dedupe node-pre-gyp (when pulling via the npm repository)
• then you might have one of the modules fail to install because node-pre-gyp is not available on PATH
  when it is needed (due to deduping)

[springmeyer]

Closing this. As long as you are running node > 4:

• You should not need to bundle deps
• You should not need to preinstall node-pre-gyp
• Just list node-pre-gyp in your deps. It should be installed early on in the process such that it can be used to install the binaries for the module it is a dependency of

If anyone finds otherwise, please comment here. But node-sqlite3 has been released recently without any problems without using preinstall or bundledDependencies, so I'm considering this issue resolved.

refs #403