Too many open files (os error 24) on macOS

[nathany]

When running lychee README.md against this repository https://github.com/dariubs/GoBooks, I am seeing the following error on macOS 13.5.1, however these links are considered fine by GitHub Actions with Ubuntu.

✗ [ERR] https://github.com/thewhitetulip/web-dev-golang-anti-textbook | Failed: Network error: dns error: proto error: io error: Too many open files (os error 24)

It's happening on quite a few links.

Running against commit 1c71e7329e369b510afb806f484aa4e415b9aeb2 of https://github.com/dariubs/GoBooks with lychee installed via Homebrew.

❯ lychee --version
lychee 0.13.0

[nathany]

Apologies, I didn't see an issue for it, but it's addressed in the README.

commit f2109a4

Still, kind've wonder if the default max concurrency should be lower, at least on macOS, to avoid this out-of-the-box issue.

[mre]

Yeah, we should fix that. I will look at the repo to see what I can do.

[mre]

Similar discussion: RustScan/RustScan#25

[nathany]

@mre Thanks. It is still occurring on main with that repo, with ~148 links to check.

Setting --max-concurrency 32 resolves it on macOS.

[mre]

• Try to use rlimit crate to detect resource limits and limit the number open files to that.

[reagle]

I just encountered this on:

| lychee 0.14.3
| macOS 14.3.1 23D60 arm64

[mre]

I think rlimit is the way to go to fix this once and for all (famous last words).
In the meantime, your best bet is --max-concurrency, unfortunately.

[mre]

We recently switched to ignore in #1500 thanks to @thomas-zahner. I just tried triggering this bug again and so far haven't managed to. Tried the dariubs/GoBooks repo and a few others. It's fast and stable. 🤞

@nathany, @reagle, if you have time to try again and report back, that would be awesome. Closing as fixed. We can reopen the issue if someone finds a way to reproduce it.