TODO: use local headers if available (was: Why do headers need to be downloaded?)

[bdharrington7]

This is an issue that has been causing me a lot of trouble lately, and it causes issues like #1230 #1231 #1232 #1234

Is there an unavoidable reason for grabbing headers from the (not 100% reliable) internet to compile a module? Isn't this just something that could be included in the node package?

[gibfahn]

I have also wondered why this is necessary, node-gyp could look for headers in the node install first, and fall back to the internet.

cc/ @bnoordhuis

[bnoordhuis]

That wouldn't work on Windows (no headers) and it's unreliable on Unices because the headers in /usr/include or /usr/local/include or wherever might not match the node binary. A simpler, robuster solution might be to simply embed the headers in the binary.

[bdharrington7]

I'm assuming here, but if node-gyp knows to fetch the node 8.1.2 headers when running on node 8.1.2 of some flavor of unix, and not to fetch them on Windows, wouldn't it be able to do the same sort of analysis and get the headers from the node binary (assuming they exist there), only if they're needed?

A simpler, robuster solution might be to simply embed the headers in the binary.

Is this feasible in node-gyp or does the onus fall on the node binary to provide that?

[bnoordhuis]

Node would first need to bake them into the binary before node-gyp can start using them. It's not out of the question but will probably need some yak shaving first.

[refack]

Is this feasible in node-gyp or does the onus fall on the node binary to provide that?

Node would first need to bake them into the binary before node-gyp can start using them. It's not out of the question but will probably need some yak shaving first.

While we're revving up the yak shearer, maybe something as simple as a more informative error message: Headers failed to download, please download manually and run with `--tarball <location>`

[bnoordhuis]

Different yak. Pull requests welcome though, I like the idea.

[rvagg]

This has been a very longstanding TODO for node-gyp. The vast majority of Node installations should have headers with them now, we did a bunch of work back in late 0.10, or io.js iirc to make it consistent. All that's needed is some magic in node-gyp to find it and use the local files instead of fetching but nobody has done that work (I started it once ...).

[andir]

Just another user here: I ran into a related problem when running npm install in a sandboxed (no network, no system libraries unless declared by me) environment where all the npm package sources are available locally. This is basically an attempt at hermetic node builds using Nix. (Same issue also applies to anyone using bazel. IIRC they are right allowing arbitrary network access to work around the issue.)
In my case some library down the road did invoke node-gyp as part of their install action. I have no control over the arguments that are passed to it without patching the sources. I think it would be desirable to have an environment variable to specify a fallback location of the node sources or the location of the tarball. Especially being able to avoid network access would be nice. I tried setting disturl in the node configuration to file:// location but that didn't work since requests doesn't seem to support that. My current (hacky) workaround was to spawn a webserver within the sandbox and redirect node-gyp (via disturl) to retrieve it from there.

I can't promise much but I might work on this issue in the coming days unless there are concerns about having an environmental (or npmrc based) fallback parameter.

EDIT: For my part it would already be enough if --ensure would be the default and thus node-gyp checking for already cached headers for the current node version. I could pre-populate the cache folder easily.

[richardlau]

I think it would be desirable to have an environment variable to specify a fallback location of the node sources or the location of the tarball. Especially being able to avoid network access would be nice.

node-gyp will convert any environment variable whose name begins with npm_config_ to the equivalent option. So you should already be able to set e.g. npm_config_nodedir or npm_config_tarball.

node-gyp/lib/node-gyp.js

Lines 136 to 154 in aaf33c3

	// support for inheriting config env variables from npm
	var npmConfigPrefix = 'npm_config_'
	Object.keys(process.env).forEach(function (name) {
	if (name.indexOf(npmConfigPrefix) !== 0) {
	return
	}
	var val = process.env[name]
	if (name === npmConfigPrefix + 'loglevel') {
	log.level = val
	} else {
	// add the user-defined options to the config
	name = name.substring(npmConfigPrefix.length)
	// gyp@741b7f1 enters an infinite loop when it encounters
	// zero-length options so ensure those don't get through.
	if (name) {
	this.opts[name] = val
	}
	}
	}, this)

[Nonius]

I try to stop the download too. I've set up npm_config_tarball where I have the headers. But it is not clear what should be the values of the npm_config_ensure environment variable. Can someone tell me? Empty? yes or true?