Exceeded available file handles on OSX. Error message should reflect this. #1044
Comments
|
I think this might be hitting a file handle limit on mac os. We have a fairly large project (one repo for many services) which means we have a large number of dependancies. I ran: And reran the update. It has now worked correctly. It seems the main issue here is the error message which does not point you at a file handle issue so it is challenging to know what is going wrong. I would argue that this is also not great for users to have to up their file handle limit. I don't know what the code is doing such that it needs so many file handles but perhaps doing a bit less in parallel is in order. |
jonomacd
changed the title
sdboyer
changed the title
|
hi, welcome! thanks for the issue. yeah, OSX has a really, ridiculously low default limit on open file handles per process (256). by far the easiest solution here is cranking that up. i suspect it would be prohibitively difficult for us to check on the ulimit being the problematic condition at the moment that an error occurs. however, it might be feasible for us to sniff the final error when dep is exiting, and if it seems like it might be ulimit related, check that value and report that the user might want to increase it. maybe. i've no idea what interacting with ulimits from code looks like 😄 |
|
It took me quite some time to diagnose and caused much frustration. The kind of frustration that puts people off and gives a bad impression. Now that I know the issue though, I'm fine with it. If it had said it was a file handle problem, I would have just upped it an been done with it. I say it would be a very good idea to try to detect this error as lots of organisations have large projects these days. Better yet, if you can increase the file limits for your process (not sure if you can do that) such that this won't happen at all, that would be the best solution. |
|
i looked around a bit on this, and found that it seems there isn't even a non-portable call that we can rely on to dynamically determine the maximum number of open file handles from within a running process. For Linux, we could sniff instead, i think we should take this as an indication that we need to institute some coarse-grained controls over the amount of concurrent behavior we allow the system to trigger in general. right now, when we trigger concurrent activity, we tend to just do it without - we'll happily kick off a goroutine for everything that we see as needing doing, and just wait until they all finish, ultimately leaving it up to the OS to decide how access to disk and network resources are meted out. rather than refactor those invocation points (which would make things a lot more complicated in a lot of places), we should grab central points of control and impose limits on the amount of concurrency (basically, goroutines) that we allow. there might be a few outliers, but i think there's really one central point of control we can use for all of this, and in doing so probably take care of most of the problem. |
UPDATE: Please see comment below. The real issue is running out of file handlers.
What version of Go (
go version) anddep(git describe --tags) are you using?go version go1.8.3 darwin/amd64
dep version v0.3.0-36-g044c4ef
What
depcommand did you run?It hung for about an hour, then I did a SIGQUIT to dump the stack:
What did you expect to see?
I expected it to update the
cloud.google.com/gopackage.What did you see instead?
The command hung for a good hour before I did a SIGQUIT to dump the stack
I have tried to update this library a number of times now. It really isn't happy about it and hanging is the most extreme example.
Other issues I have seen trying to update
cloud.google.com/go:Then running it again straight after:
After this the
$GOPATH/pkg/dep/is in a bad state. Many directories have all of the go files deleted. For example:These deletes have happened to loads of files in the dep folder.
To me, this smells like a network issue with some bad network error handling in dep. The trouble is I don't think I have a bad network connection. All other network operations I do appear to be fine. I am not sure why
http://go4.org?go-get=1seems unhappy with me.After deleting the
$GOPATH/pkg/depfolder and trying again:And finally running it again gives the no host found error (Sorry for wall of text, I want to give you as much detail as possible):
It seems to oscillate between a network error and no buildable go code while it fully hung one time. I still don't know how to update my package. Any help would be appreciated.
Thanks!
The text was updated successfully, but these errors were encountered: