dial tcp 127.0.0.1:8000: socket: too many open files

[liziqiang0523]

Bug 描述
当 同时启动的任务多 时，爬虫启动全部失败，报 62fddbdd4f6290e5182ab109/oreo/spiders/henan/init.py": dial tcp 127.0.0.1:8000: socket: too many open files错误。

复现步骤
该 Bug 复现步骤如下

1. 同时启动很多爬虫，例如同时启动30个scrapy项目。
2. 观察是否有启动错误。

期望结果
希望能正常调度爬虫任务，多的时候也是一样。

截屏

[liziqiang0523]

crawlab_master | [GIN] 2022/09/28 - 16:00:07 | 200 | 977.422µs | 127.0.0.1 | GET "/filer/fs/62fde0024f6290e5182abdbe/oreo/spiders/shanghai/init.py"
crawlab_master | 2022/09/28 16:00:07 info handle: {{/root/crawlab_workspace/62fde0024f6290e5182abdbe/oreo/spiders/shanghai/parser_shggzy.py /fs/62fde0024f6290e5182abdbe/oreo/spiders/shanghai/parser_shggzy.py false map[] []} 0x9c6be0 0xc217007e00}
crawlab_master | open /root/crawlab_workspace/62fde0024f6290e5182abdbe/oreo/spiders/shanghai/init.py: too many open files
crawlab_master | /go/pkg/mod/github.com/crawlab-team/go-trace@v0.1.1/trace.go:13 github.com/crawlab-team/go-trace.TraceError()
crawlab_master | /go/pkg/mod/github.com/crawlab-team/crawlab-fs@v0.6.0-beta.20211101.1940.0.20220912130912-a47819e0c7c9/seaweedfs_manager.go:445 github.com/crawlab-team/crawlab-fs.(*SeaweedFsManager).downloadFile.func1()
crawlab_master | /go/pkg/mod/github.com/crawlab-team/goseaweedfs@v0.6.0-beta.20211101.1936.0.20220912021203-dfee5f74dd69/http_client.go:155 github.com/crawlab-team/goseaweedfs.(*httpClient).download()
crawlab_master | /go/pkg/mod/github.com/crawlab-team/goseaweedfs@v0.6.0-beta.20211101.1936.0.20220912021203-dfee5f74dd69/filer.go:209 github.com/crawlab-team/goseaweedfs.(*Filer).Download()
crawlab_master | /go/pkg/mod/github.com/crawlab-team/crawlab-fs@v0.6.0-beta.20211101.1940.0.20220912130912-a47819e0c7c9/seaweedfs_manager.go:422 github.com/crawlab-team/crawlab-fs.(*SeaweedFsManager).downloadFile()
crawlab_master | /go/pkg/mod/github.com/crawlab-team/crawlab-fs@v0.6.0-beta.20211101.1940.0.20220912130912-a47819e0c7c9/seaweedfs_manager.go:310 github.com/crawlab-team/crawlab-fs.(*SeaweedFsManager).start.func1.1()
crawlab_master | /go/pkg/mod/github.com/cenkalti/backoff/v4@v4.1.0/retry.go:55 github.com/cenkalti/backoff/v4.RetryNotifyWithTimer()
crawlab_master | /go/pkg/mod/github.com/cenkalti/backoff/v4@v4.1.0/retry.go:34 github.com/cenkalti/backoff/v4.RetryNotify()
crawlab_master | /go/pkg/mod/github.com/cenkalti/backoff/v4@v4.1.0/retry.go:28 github.com/cenkalti/backoff/v4.Retry()
crawlab_master | /go/pkg/mod/github.com/crawlab-team/crawlab-fs@v0.6.0-beta.20211101.1940.0.20220912130912-a47819e0c7c9/seaweedfs_manager.go:309 github.com/crawlab-team/crawlab-fs.(*SeaweedFsManager).start.func1()
crawlab_master | /usr/local/go/src/runtime/asm_amd64.s:1371 runtime.goexit()
crawlab_master | [GIN] 2022/09/28 - 16:00:07 | 200 | 1.007319ms | 127.0.0.1 | GET "/filer/fs/62fddf344f6290e5182abbb3/oreo/spiders/tianjin/node_modules/crypto-js/hmac-ripemd160.js"
crawlab_master | open /root/crawlab_workspace/62fddf344f6290e5182abbb3/oreo/spiders/tianjin/node_modules/crypto-js/hmac-ripemd160.js: too many open files
crawlab_master | /go/pkg/mod/github.com/crawlab-team/go-trace@v0.1.1/trace.go:13 github.com/crawlab-team/go-trace.TraceError()
crawlab_master | /go/pkg/mod/github.com/crawlab-team/crawlab-fs@v0.6.0-beta.20211101.1940.0.20220912130912-a47819e0c7c9/seaweedfs_manager.go:445 github.com/crawlab-team/crawlab-fs.(*SeaweedFsManager).downloadFile.func1()
crawlab_master | /go/pkg/mod/github.com/crawlab-team/goseaweedfs@v0.6.0-beta.20211101.1936.0.20220912021203-dfee5f74dd69/http_client.go:155 github.com/crawlab-team/goseaweedfs.(*httpClient).download()
crawlab_master | /go/pkg/mod/github.com/crawlab-team/goseaweedfs@v0.6.0-beta.20211101.1936.0.20220912021203-dfee5f74dd69/filer.go:209 github.com/crawlab-team/goseaweedfs.(*Filer).Download()
crawlab_master | /go/pkg/mod/github.com/crawlab-team/crawlab-fs@v0.6.0-beta.20211101.1940.0.20220912130912-a47819e0c7c9/seaweedfs_manager.go:422 github.com/crawlab-team/crawlab-fs.(*SeaweedFsManager).downloadFile()
crawlab_master | /go/pkg/mod/github.com/crawlab-team/crawlab-fs@v0.6.0-beta.20211101.1940.0.20220912130912-a47819e0c7c9/seaweedfs_manager.go:310 github.com/crawlab-team/crawlab-fs.(*SeaweedFsManager).start.func1.1()
crawlab_master | /go/pkg/mod/github.com/cenkalti/backoff/v4@v4.1.0/retry.go:55 github.com/cenkalti/backoff/v4.RetryNotifyWithTimer()
crawlab_master | /go/pkg/mod/github.com/cenkalti/backoff/v4@v4.1.0/retry.go:34 github.com/cenkalti/backoff/v4.RetryNotify()
crawlab_master | /go/pkg/mod/github.com/cenkalti/backoff/v4@v4.1.0/retry.go:28 github.com/cenkalti/backoff/v4.Retry()
crawlab_master | /go/pkg/mod/github.com/crawlab-team/crawlab-fs@v0.6.0-beta.20211101.1940.0.20220912130912-a47819e0c7c9/seaweedfs_manager.go:309 github.com/crawlab-team/crawlab-fs.(*SeaweedFsManager).start.func1()
crawlab_master | /usr/local/go/src/runtime/asm_amd64.s:1371 runtime.goexit()
crawlab_master | 2022/09/28 16:00:07 info handle: {{/root/crawlab_workspace/62fddf344f6290e5182abbb3/oreo/spiders/tianjin/node_modules/crypto-js/hmac-sha1.js /fs/62fddf344f6290e5182abbb3/oreo/spiders/tianjin/node_modules/crypto-js/hmac-sha1.js false map[] []} 0x9c6be0 0xc2170c4ea0}
crawlab_master | 2022/09/28 16:00:07 info handle: {{/root/crawlab_workspace/62fde0024f6290e5182abdbe/oreo/spiders/shanghai/shggzy.py /fs/62fde0024f6290e5182abdbe/oreo/spiders/shanghai/shggzy.py false map[] []} 0x9c6be0 0xc2170423c0}
crawlab_master | 2022/09/28 16:00:07 info handle: {{/root/crawlab_workspace/62fddf344f6290e5182abbb3/oreo/spiders/tianjin/node_modules/crypto-js/hmac-sha224.js /fs/62fddf344f6290e5182abbb3/oreo/spiders/tianjin/node_modules/crypto-js/hmac-sha224.js false map[] []} 0x9c6be0 0xc2170c51a0}
crawlab_master | [GIN] 2022/09/28 - 16:00:07 | 200 | 1.331662ms | 127.0.0.1 | GET "/filer/fs/62fde0024f6290e5182abdbe/oreo/spiders/shanghai/parser_shggzy.py"
crawlab_master | open /root/crawlab_workspace/62fde0024f6290e5182abdbe/oreo/spiders/shanghai/parser_shggzy.py: too many open files^CERROR: Aborting.

[tikazyq]

Seems like the issue occurred again