go-reuseport needs to start using singlepoll on OSX

[mitar]

Version information:

go-ipfs version: 0.4.5-
Repo version: 5
System version: amd64/darwin
Golang version: go1.7.4

Type: Bug

Priority: P3

Description:

It seems there are still issues with too many open files on Mac OS X. It increases the limit initially to 1024, but after a day of running my console looks like:

Initializing daemon...
Adjusting current ulimit to 1024...
Successfully raised file descriptor limit to 1024.
Swarm listening on /ip4/127.0.0.1/tcp/4001
Swarm listening on /ip4/192.168.0.112/tcp/4001
Swarm listening on /ip6/::1/tcp/4001
API server listening on /ip4/127.0.0.1/tcp/5001
Gateway (readonly) server listening on /ip4/127.0.0.1/tcp/8080
Daemon is ready
21:57:18.979 ERROR     flatfs: too many open files, retrying in 100ms flatfs.go:180
21:57:19.235 ERROR     flatfs: too many open files, retrying in 100ms flatfs.go:180
21:57:19.336 ERROR     flatfs: too many open files, retrying in 200ms flatfs.go:180
21:57:19.536 ERROR     flatfs: too many open files, retrying in 300ms flatfs.go:180
21:57:19.837 ERROR     flatfs: too many open files, retrying in 400ms flatfs.go:180
21:57:20.237 ERROR     flatfs: too many open files, retrying in 500ms flatfs.go:180
21:57:20.738 ERROR     flatfs: too many open files, retrying in 600ms flatfs.go:180
21:57:21.542 ERROR     flatfs: too many open files, retrying in 100ms flatfs.go:180
21:57:21.643 ERROR     flatfs: too many open files, retrying in 200ms flatfs.go:180
21:57:21.844 ERROR     flatfs: too many open files, retrying in 300ms flatfs.go:180
22:18:12.873 ERROR   floodsub: already have connection to peer: <peer.ID Zewdrm> floodsub.go:112
22:33:15.290 ERROR   floodsub: already have connection to peer: <peer.ID Zewdrm> floodsub.go:112
23:12:05.159 ERROR       mdns: mdns lookup error: failed to bind to any unicast udp port mdns.go:135
23:56:44.222 ERROR     flatfs: too many open files, retrying in 100ms flatfs.go:180
23:56:44.322 ERROR     flatfs: too many open files, retrying in 200ms flatfs.go:180
23:56:45.159 ERROR       mdns: mdns lookup error: failed to bind to any multicast udp port mdns.go:135
23:56:45.216 ERROR     flatfs: too many open files, retrying in 100ms flatfs.go:180
23:56:45.316 ERROR     flatfs: too many open files, retrying in 200ms flatfs.go:180
23:56:45.517 ERROR     flatfs: too many open files, retrying in 300ms flatfs.go:180
23:56:45.837 ERROR     flatfs: too many open files, retrying in 400ms flatfs.go:180
23:56:46.237 ERROR     flatfs: too many open files, retrying in 500ms flatfs.go:180
23:56:46.739 ERROR     flatfs: too many open files, retrying in 600ms flatfs.go:180
23:56:47.541 ERROR     flatfs: too many open files, retrying in 100ms flatfs.go:180
23:56:47.642 ERROR     flatfs: too many open files, retrying in 200ms flatfs.go:180
04:48:13.779 ERROR   floodsub: already have connection to peer: <peer.ID Zewdrm> floodsub.go:112
05:53:15.159 ERROR       mdns: mdns lookup error: failed to bind to any unicast udp port mdns.go:135
07:23:06.270 ERROR   floodsub: already have connection to peer: <peer.ID Zewdrm> floodsub.go:112
07:48:15.159 ERROR       mdns: mdns lookup error: failed to bind to any multicast udp port mdns.go:135
08:23:07.536 ERROR   floodsub: already have connection to peer: <peer.ID Zewdrm> floodsub.go:112
08:38:53.221 ERROR     flatfs: too many open files, retrying in 100ms flatfs.go:180
08:38:53.322 ERROR     flatfs: too many open files, retrying in 200ms flatfs.go:180
08:38:53.523 ERROR     flatfs: too many open files, retrying in 300ms flatfs.go:180
08:38:53.823 ERROR     flatfs: too many open files, retrying in 400ms flatfs.go:180
08:38:54.224 ERROR     flatfs: too many open files, retrying in 500ms flatfs.go:180
08:38:54.725 ERROR     flatfs: too many open files, retrying in 600ms flatfs.go:180
08:38:55.159 ERROR       mdns: mdns lookup error: failed to bind to any unicast udp port mdns.go:135
08:38:55.527 ERROR     flatfs: too many open files, retrying in 100ms flatfs.go:180
08:38:55.627 ERROR     flatfs: too many open files, retrying in 200ms flatfs.go:180
08:38:55.827 ERROR     flatfs: too many open files, retrying in 300ms flatfs.go:180
08:38:56.128 ERROR     flatfs: too many open files, retrying in 400ms flatfs.go:180
08:38:56.529 ERROR     flatfs: too many open files, retrying in 500ms flatfs.go:180
08:57:35.159 ERROR       mdns: mdns lookup error: failed to bind to any unicast udp port mdns.go:135
09:28:13.902 ERROR blockstore: blockstore.AllKeysChan got err:%!(EXTRA *errors.errorString=walk failed: open /Users/mitar/.ipfs/blocks: too many open files) blockstore.go:197
09:28:15.159 ERROR       mdns: mdns lookup error: failed to bind to any unicast udp port mdns.go:135

I ran ipfs repo stat and I got strange results:

NumObjects 	 0
RepoSize 	 1916132242
RepoPath 	 /Users/user/.ipfs
Version 	 fs-repo@5

[whyrusleeping]

@mitar What is your ipfs nodes workload like? Are you running lots of ipfs adds?

[mitar]

Mostly object puts and gets, yes. I would say many more gets than puts.

Also the storage is growing like crazy.

$ ipfs stats repo
NumObjects 	 996591
RepoSize 	 12267396749
RepoPath 	 /Users/user/.ipfs
Version 	 fs-repo@5

No way I am creating so many objects? My current test blockchain has only 1000 blocks or so:

$ ipfs pin ls | wc -l
     645

[mitar]

I checked. I am getting such errors when I am doing mostly many many object gets through the HTTP API:

21:31:28.933 ERROR     flatfs: too many open files, retrying in 100ms flatfs.go:180
21:31:29.034 ERROR     flatfs: too many open files, retrying in 200ms flatfs.go:180
21:31:30.527 ERROR     flatfs: too many open files, retrying in 100ms flatfs.go:180
21:31:30.527 ERROR     flatfs: too many open files, retrying in 100ms flatfs.go:180
21:31:30.627 ERROR     flatfs: too many open files, retrying in 200ms flatfs.go:180
21:31:30.627 ERROR     flatfs: too many open files, retrying in 200ms flatfs.go:180
21:31:30.827 ERROR     flatfs: too many open files, retrying in 300ms flatfs.go:180
21:31:30.827 ERROR     flatfs: too many open files, retrying in 300ms flatfs.go:180
21:31:41.896 ERROR     flatfs: too many open files, retrying in 100ms flatfs.go:180
21:31:46.185 ERROR     flatfs: too many open files, retrying in 100ms flatfs.go:180
21:31:46.286 ERROR     flatfs: too many open files, retrying in 200ms flatfs.go:180
21:31:46.486 ERROR     flatfs: too many open files, retrying in 300ms flatfs.go:180
21:31:46.787 ERROR     flatfs: too many open files, retrying in 400ms flatfs.go:180
21:31:47.187 ERROR     flatfs: too many open files, retrying in 500ms flatfs.go:180
21:32:25.471 ERROR     flatfs: too many open files, retrying in 100ms flatfs.go:180
21:32:27.648 ERROR     flatfs: too many open files, retrying in 100ms flatfs.go:180
21:32:27.750 ERROR     flatfs: too many open files, retrying in 200ms flatfs.go:180

[whyrusleeping]

hey @mitar, could you try getting the debug info i mention here: #3763 (comment) ?

[mitar]

Attached.

Also, it seems not all "too many open files" are detected by IPFS. I also get ECONNRESET errors on my node (HTTP JS API) side when it reaches the maximum.

files.txt

[whyrusleeping]

@Kubuxu hrm... seems we need to fix reuseport on OSX finally (or bump the default fd limit to buy us time)

[whyrusleeping]

@mitar As a workaround until we get this addressed better, you can set the env var IPFS_FD_MAX to something like 4096.

[mitar]

I created this repository for one other reproduction, but if I start ipfs daemon and soon afterwards run the script, it often disconnects the script during run because daemon reaches file descriptor limit.

[Kubuxu]

There is hope for go-reuseport situation getting resolved as golang/go#9661 (comment) has seen some progress.

[Stebalien]

(no longer an issue)