npm 2.5 and 2.6 fails with getaddrinfo EAI_FAIL when only loopback IP available

[msimerson]

# uname -a
FreeBSD smtp 10.1-RELEASE-p6 FreeBSD 10.1-RELEASE-p6 #0: Tue Feb 24 19:00:21 UTC 2015     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64

[root@smtp ~]# ifconfig   
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
    ether 00:0c:29:09:12:e8
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    inet 127.0.0.9 netmask 0xffffffff 

# npm install -g ws
npm ERR! FreeBSD 10.1-RELEASE-p6
npm ERR! argv "node" "/usr/local/bin/npm" "install" "-g" "ws"
npm ERR! node v0.12.0
npm ERR! npm  v2.6.0
npm ERR! code EAI_FAIL
npm ERR! errno EAI_FAIL
npm ERR! syscall getaddrinfo

npm ERR! getaddrinfo EAI_FAIL
npm ERR! 
npm ERR! If you need help, you may report this error at:
npm ERR!     <http://github.com/npm/npm/issues>

npm ERR! Please include the following file with any support request:
npm ERR!     /root/npm-debug.log
[root@smtp ~]# cat npm-debug.log 
0 info it worked if it ends with ok
1 verbose cli [ 'node', '/usr/local/bin/npm', 'install', '-g', 'ws' ]
2 info using npm@2.6.0
3 info using node@v0.12.0
4 silly cache add args [ 'ws', null ]
5 verbose cache add spec ws
6 silly cache add parsed spec { raw: 'ws',
6 silly cache add   scope: null,
6 silly cache add   name: 'ws',
6 silly cache add   rawSpec: '',
6 silly cache add   spec: '*',
6 silly cache add   type: 'range' }
7 verbose addNamed ws@*
8 silly addNamed semver.valid null
9 silly addNamed semver.validRange *
10 silly addNameRange { name: 'ws', range: '*', hasData: false }
11 silly mapToRegistry name ws
12 silly mapToRegistry using default registry
13 silly mapToRegistry registry https://registry.npmjs.org/
14 silly mapToRegistry uri https://registry.npmjs.org/ws
15 verbose addNameRange registry:https://registry.npmjs.org/ws not in flight; fetching
16 verbose request uri https://registry.npmjs.org/ws
17 verbose request no auth needed
18 info attempt registry request try #1 at 14:34:44
19 verbose request id ec4a2d4d11e53a85
20 http request GET https://registry.npmjs.org/ws
21 info retry will retry, error on last attempt: Error: getaddrinfo EAI_FAIL
22 info attempt registry request try #2 at 14:34:54
23 http request GET https://registry.npmjs.org/ws
24 info retry will retry, error on last attempt: Error: getaddrinfo EAI_FAIL
25 info attempt registry request try #3 at 14:35:54
26 http request GET https://registry.npmjs.org/ws
27 verbose stack Error: getaddrinfo EAI_FAIL
27 verbose stack     at Object.exports._errnoException (util.js:746:11)
27 verbose stack     at errnoException (dns.js:49:15)
27 verbose stack     at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:94:26)
28 verbose cwd /root
29 error FreeBSD 10.1-RELEASE-p6
30 error argv "node" "/usr/local/bin/npm" "install" "-g" "ws"
31 error node v0.12.0
32 error npm  v2.6.0
33 error code EAI_FAIL
34 error errno EAI_FAIL
35 error syscall getaddrinfo
36 error getaddrinfo EAI_FAIL
37 error If you need help, you may report this error at:
37 error     <http://github.com/npm/npm/issues>
38 verbose exit [ 1, true ]

[root@smtp ~]# cat /etc/resolv.conf
nameserver 127.0.0.3

[root@smtp ~]# host registry.npmjs.org
registry.npmjs.org is an alias for a.sni.fastly.net.
a.sni.fastly.net has address 23.235.47.162

[root@smtp ~]# curl -O https://registry.npmjs.org/ws
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 70682  100 70682    0     0  67821      0  0:00:01  0:00:01 --:--:-- 67833

I've tried installing iojs 1.2 instead but get the same error when trying to do a npm install.

Having only a single loopback address is because this FreeBSD instance is actually a FreeBSD jail running under VMWare on my Mac. The jail host is running PF with NAT and provides each of the jails with networking:

nat on em0 from 127.0.0.0/8 to any -> (em0)

This failure is something new in the way node is now handling DNS, as this was working just fine with node 0.10 and below. As I showed with host and curl above, DNS and networking are obviously working. In fact, they work just fine for all these other apps:

root@freebsd10:~ # jls
   JID  IP Address      Hostname                      Path
     1  127.0.0.4       mysql                         /jails/mysql
     3  127.0.0.5       clamav                        /jails/clamav
     6  127.0.0.12      haproxy                       /jails/haproxy
     8  127.0.0.6       spamassassin                  /jails/spamassassin
     9  127.0.0.3       dns                           /jails/dns
    10  127.0.0.7       dspam                         /jails/dspam
    11  127.0.0.10      webmail                       /jails/webmail
    17  127.0.0.9       smtp                          /jails/smtp

Here's some tcpdump output with more interesting info.

[smikes]

This is definitely a bug, but the problem is, I'm not sure where to start looking to track it down.

I believe the underlying problem is with the request module or with node itself, since npm does not directly handle any https traffic itself. I'm looking through the code and I don't see getaddrinfo being explicitly called by any javascript under npm itself.

Are you building node yourself or are you downloading a pre-made version? I would be curious to see if you can run the node https integration tests with your setup.

[smikes]

It looks like FreeBSD might have different error handling from Linux hosts; where Linux getaddrinfo() returns EAI_NONAME, FreeBSD returns EAI_FAIL (see this comment from samba: https://bugzilla.samba.org/attachment.cgi?id=8563&action=edit )

[msimerson]

I'm not sure where to start looking

That makes two of us. At first I figured the bug might be in dns.js, but in poking around in there, I'm not so sure.

[smikes]

No, it definitely is dns.js. I think it must have landed with this commit: nodejs/node-v0.x-archive@ca9eb71

and where EAI_NONAME, EAI_NODATA are being handled, a case for EAI_FAIL should be added too.

Can you build iojs in your FreeBSD image, or is that untenable?

[smikes]

My guess as to the offending line: nodejs/node-v0.x-archive@ca9eb71#diff-4cce137cb8157852360b2884ec408341R34

[smikes]

I'm going to open a case on iojs to see if they have any ideas.

[smikes]

I asked at io.js and they pointed out where npm is expecting ENOTFOUND and not handling EAI_FAIL -- that's in lib/utils/error_handler.js We can definitely improve that error message.

Unfortunately that's not going to fix the problem for you, @msimerson, since the problem for you is that node is getting EAI_FAIL even when host and other commands succeed with your DNS setup.

Do you have a docker image of this FreeBSD setup that I could use for testing?

[msimerson]

Hmm, I wasn't aware that one could run FreeBSD under Docker. Google wasn't helpful there, got a link on how to do that? The "test" setup in this case is a Mac running VMware 7 with a stock FreeBSD 10.1 amd64 installed. Then I create a jail with a loopback address, turn on PF with that NAT rule shown above, fire up the jail and install node with pkg install node.

[smikes]

Sorry, I am speaking without knowledge. I mean, is there a way you could provide someone unskilled w/ virtual machines (me) an environment where I can reproduce this?

On Sun, Mar 1, 2015 at 10:18 AM, Matt Simerson notifications@github.com
wrote:

Hmm, I wasn't aware that one could run FreeBSD under Docker. Google wasn't helpful there, got a link on how to do that? The "test" setup in this case is a Mac running VMware 7 with a stock FreeBSD 10.1 amd64 installed. Then I create a jail with a loopback address, turn on PF with that NAT rule shown above, fire up the jail and install node with pkg install node.

Reply to this email directly or view it on GitHub:
#7508 (comment)

[msimerson]

I'll see if I can't do that. Do you have a public SSH key I can install for you to access it? Thanks for all your effort to provide a resolution for this.

[othiym23]

bcd4722 adds this to the error handler, and I think handles the bit that is relevant to npm. Feel free to continue discussing this here, but I'm closing the issue, as it's a joyent/node / iojs/io.js problem with FreeBSD, looks like.

BTW, I like the sound of small VirtualBox images containing vms exhibiting particular problems, even though the logistics seem a little challenging. If people have access to AWS, we could just pass them around as AMIs, maybe?

[saper]

I bumped into this too today. Looks strange to me - @msimerson, were you using FreeBSD node port?
It fails for me also within a poudriere jail even if I have a full set of IP addresses available:

Current FreeBSD node port has a little patch to lib/dns.js:

https://svnweb.freebsd.org/ports/head/www/node/files/patch-lib_dns.js?revision=378937&view=markup

@msimerson do you have this patch in your node? cc: @pct

[msimerson]

were you using FreeBSD node port?

Yes.

do you have this patch in your node?

Yes. I can confirm that it doesn't fix the problem. See iojs #337, iojs #1011, node #8540.

The state in current ports hasn't improved yet.

[root@smtp ~]# node http-get.js 
Got response: 200
[root@smtp ~]# pkg delete node010
[smtp] [1/1] Deleting files for node010-0.10.38: 100%

[root@smtp ~]# pkg install node
[smtp] [1/1] Extracting node-0.12.0_1: 100%
[root@smtp ~]# node http-get.js 
Got error: getaddrinfo EAI_FAIL

[root@smtp ~]# pkg delete node
[smtp] [1/1] Deleting files for node-0.12.0_1: 100%
[root@smtp ~]# pkg install iojs
[smtp] [1/1] Extracting iojs-1.4.3: 100%
[root@smtp ~]# node http-get.js 
Got error: getaddrinfo EAI_FAIL www.google.com:80

require('http').get("http://www.google.com/", function(res) {
  console.log("Got response: " + res.statusCode);
}).on('error', function(e) {
  console.log("Got error: " + e.message);
});

[saper]

What does node 0.12 say if you ask for os.networkInterfaces() ?

[msimerson]

[root@smtp ~]# pkg install -y node
FreeBSD repository is up-to-date.
[smtp] [1/1] Extracting node-0.12.0_1: 100%
[root@smtp ~]# node
> os.networkInterfaces()
{ lo0: 
   [ { address: '127.0.0.9',
       netmask: '255.255.255.255',
       family: 'IPv4',
       mac: '00:00:00:00:00:00',
       internal: true } ] }
> 

Which is quite accurate.

# ifconfig
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
    ether 00:0c:29:09:12:e8
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    inet 127.0.0.9 netmask 0xffffffff 

Networking is handled via PF rules outside the jail.

# default route to the internet for jails on loopback IPs
nat on $ext_if from 127.0.0.1/8 to any -> ($ext_if)

[saper]

And /etc/resolv.conf?

[msimerson]

% cat /etc/resolv.conf
nameserver 127.0.0.3

(A few jails over, unbound is listening on 127.0.0.3.)

[baznikin]

I suppose the case is in address from 127/8 space. I tried to bind jail on my real interface (em0 instead of lo0) without success. Then I change address to 192.168.1.1 and it works like a charm

[niklaas]

Giving my jail an 192.168.* address fixed this for me too.

[knachev]

I can confirm that giving my jail an 192.168.* address fixed this for me too. Thanks!