-
Notifications
You must be signed in to change notification settings - Fork 153
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
mac: runtime: failed to create new OS thread #554
Comments
Hi, a quick web search for "runtime: failed to create new OS thread golang m1 mac" gives me several mentions of people having similar trouble with docker- are you using docker here too? Is that the entire log line, or does it have something like If you are using docker, I suggest trying the bazel-remote-2.3.7-darwin-arm64 binary from https://github.com/buchgr/bazel-remote/releases without docker. |
Hi, @mostynb no we are not using docker. And yes, that is the entire log line. There is nothing else before or after that. I even tried with latest merges (built one locally with latest changes) and still the issue persists. I am pretty certain this issue started happening when we reached the storage limit which we have set for cache. I was hoping #480 would fix it but issue still persists. Somehow, it is not able to cleanup the files as fast as cache is being is written. For e.g. In above screenshot, you can see current cache size is around 400GB but our limit is set to 375GB. First crash happened when we crossed initial 300GB limit, hence we had increased it to 375GB it crossed that limit too. |
Is this a HDD or an SSD? |
@mostynb its an SSD |
bazel-remote doesn't scan the data dir every time the disk usage metrics are updated (because that's really slow) - if you manually remove files from the data dir the metrics will be wrong until bazel-remote is restarted. I wonder if you can try restarting bazel-remote with a smaller size limit, say 300G? |
That's what I figured. We already had a limit of 300GB and it went above that. Hence, I had to increase the limit. |
bazel-remote only uses an approximation of the actual disk space used (because some is lost due to filesystem overheads, which can vary between filesystems). If you set a size limit and bazel-remote let the actual usage go too far over that, then you should reduce bazel-remote's limit (not increase it). Do you run anything else on this mac mini? Is it close to running out of space on the SSD? |
Reducing the limit introduced the mentioned error in first place. I reduced the limit from 400GB to 300GB because cache was way above the limit and I thought reducing the limit will clean up the LRU cache aggressively. Now we are at around 340GB and limit is set to 375GB and haven't seen the mentioned error yet, though I assume it will come once the size passes the set limit of 375GB. |
Looking into the go runtime source makes me think this might be a mac-specific limitation, like the one mentioned here: I wonder if you could try enabling performance mode on your mac mini and see if that helps? https://support.apple.com/en-us/HT202528 |
These threads are about mac-os server and we are running vanilla version of mac-os. Still tried running the suggested commands and it gives error. Since, we want to increase threads per_process, will check if there is any other way to to this. |
Ohk, managed to increase the limit by using some custom plists. Right now cache is at 340GB and limit is set to 375GB, so its not going to crash anytime soon. Will decrease the limit to 300 on weekend and test as currently builds are using this cache. If it successfully comes down to less than 300 or even near to 300 then we can assume the fix worked else will have to debug more. |
@mostynb CMIIW, according to https://github.com/buchgr/bazel-remote/blob/master/cache/disk/disk.go#L142 , bazel-remote might spawn 5000 threads to cleanup the disk in worst case scenario. Since this value is still more than allowed for a user (atleast in macOS), this might be the reason it is crashing for us. If it is so, will find a way to set this dynamically. |
Update: I reduced the limit to 300 and it failed again.
|
That's approximately correct, yes. bazel-remote attempts to create no more than 5,000 simultaneous file removal operations, because Go has a 10,000 os thread limit and each file removal might block and cause the creation of another os thread. Besides those potentially 5,000 blocks os threads, there will be some other os threads used by bazel-remote (but hopefully many fewer than 5k). Your most recent log shows the crash occurring when bazel-remote was still building the LRU index. Since you have reduced your cache limit, I suspect that bazel-remote was removing a bunch of files, and hit some limit that is less than 5000 os threads. So I think we'll need to do a bit of research to figure out what that limit is on your machine, and how we might be able to raise it. This also means that your disk is having trouble keeping up with all these file removals. Is this an intel mac mini or an m1 mac mini? Which version of Mac OS is it running? Can you share the output from |
This is the output of launctl limit. I tried raising maxproc to more than 4000 but couldn't. I was also looking into a way to set the semaphore dynamically instead of just hardcoding it, but couldn't find proper syscall to make. |
Interesting- what is the semaphore setting, and how did you change it? |
The line I was talking about. shubhindia@8a072cb |
This also adds a startup log line mentioning the value, to help debugging. This might avoid buchgr#554.
Oh, right. If this works for you, how about we lower this to 3000 only for mac, and add a log line for debugging: #556 ? |
This also adds a startup log line mentioning the value, to help debugging. This might avoid buchgr#554.
Closing this as setting the semaphore limit to lower value fixes the issue. |
We are using latest release available.
We are using bazel-remote as a cache backend for ios-builds. It was running fine for about 3 weeks now but now we are getting below error after starting the server.
There aren't any logs available apart from the access logs. Only above error. We are running it on an m1-mac-mini with 512 GB
Host OS: Mac OS
Version: Mac OS 12.2.1
RAM: 16GB
The text was updated successfully, but these errors were encountered: