Extreme parallelism used for subcommands bombs Raspberry Pi out of memory

[danielbprice]

What version of dep are you using (dep version)?

v0.4.1-34-g42f17824

What dep command did you run?

dep ensure -v -update on a Raspberry Pi Model 3 with 1GB of RAM. dep status for my project has 61 lines of output.

What did you expect to see?

I expected to see dep ensure -v update finish in a reasonable amount of time. On my x86 systems, the runtime is about 24 seconds. On the raspberry pi model 3, we're into 20 minutes+ territory if the system starts paging, or else the system runs out of memory completely.

What did you see instead?

Super duper high load averages on the Raspberry pi, along with extremely high memory usage, resulting in paging, or total memory exhaustion.

This seems to be happening because dep is launching many dozens of git commands in parallel without regard for load on the machine or the size. My project has exceeded the size where this works.

Here is an example from pstree wherein I caught dep in the middle of such a scan:

      |   `-sh -c cd /ex0/home/dp/Product/golang/src/bg && /ex0/home/dp/Product/golang/bin/dep ensure -v -update
      |       `-dep ensure -v -update
      |           |-git fetch --tags --prune origin
      |           |-(git)
      |           |-git fetch --tags --prune origin
      |           |   `-(git-remote-http)
      |           |-git fetch --tags --prune origin
      |           |   `-(git-remote-http)
      |           |-git fetch --tags --prune origin
      |           |   `-(git-remote-http)
      |           |-git fetch --tags --prune origin
      |           |   `-git-remote-http origin https://github.com/__redacted__
      |           |-git fetch --tags --prune origin
      |           |   `-git-remote-http origin https://github.com/__redacted__
      |           |-git fetch --tags --prune origin
      |           |   `-git-remote-http origin https://github.com/__redacted__
      |           |-git fetch --tags --prune origin
      |           |   `-git-remote-http origin https://github.com/__redacted__
      |           |-git fetch --tags --prune origin
      |           |   `-git-remote-http origin https://github.com/__redacted__
      |           |-git fetch --tags --prune origin
      |           |   `-git-remote-http origin https://github.com/__redacted__
      |           |-git fetch --tags --prune origin
      |           |   `-git-remote-http origin https://github.com/__redacted__
      |           |-git fetch --tags --prune origin
      |           |   `-git-remote-http origin https://github.com/__redacted__
      |           |-git fetch --tags --prune origin
      |           |   `-git-remote-http origin https://github.com/__redacted__
      |           |-git fetch --tags --prune origin
      |           |   `-git-remote-http origin https://go.googlesource.com/net
      |           |-git fetch --tags --prune origin
      |           |   `-git-remote-http origin https://github.com/__redacted__
      |           |-git fetch --tags --prune origin
      |           |   `-git-remote-http origin https://github.com/__redacted__
      |           |-git fetch --tags --prune origin
      |           |   `-git-remote-http origin https://github.com/__redacted__
      |           |-git fetch --tags --prune origin
      |           |   `-git-remote-http origin https://github.com/__redacted__
      |           |-git fetch --tags --prune origin
      |           |   `-git-remote-http origin https://github.com/__redacted__
      |           |-git fetch --tags --prune origin
      |           |   `-git-remote-http origin https://github.com/__redacted__
      |           |-git fetch --tags --prune origin
      |           |   `-git-remote-http origin https://go.googlesource.com/text
      |           |-git fetch --tags --prune origin
      |           |   `-git-remote-http origin https://github.com/__redacted__
      |           |-git fetch --tags --prune origin
      |           |   `-git-remote-http origin https://github.com/__redacted__
      |           |-git fetch --tags --prune origin
      |           |   `-git-remote-http origin https://go.googlesource.com/oauth2
      |           |-git fetch --tags --prune origin
      |           |   `-git-remote-http origin https://github.com/__redacted__
      |           |-git fetch --tags --prune origin
      |           |   `-git-remote-http origin https://go.googlesource.com/crypto
      |           |-git fetch --tags --prune origin
      |           |   `-git-remote-http origin https://code.googlesource.com/google-api-go-client
      |           |-git ls-remote https://github.com/__redacted__
      |           |   `-git-remote-http https://github.com/__redacted__ https://github.com/__redacted__
      |           |-git ls-remote https://github.com/__redacted__
      |           |   `-git-remote-http https://github.com/__redacted__ https://github.com/__redacted__
      |           |-git ls-remote https://github.com/__redacted__
      |           |   `-git-remote-http https://github.com/__redacted__ https://github.com/__redacted__
      |           |-git ls-remote https://github.com/__redacted__
      |           |   `-git-remote-http https://github.com/__redacted__ https://github.com/__redacted__
      |           |-git ls-remote https://github.com/__redacted__
      |           |   `-git-remote-http https://github.com/__redacted__ https://github.com/__redacted__
      |           |-git ls-remote https://github.com/__redacted__
      |           |   `-git-remote-http https://github.com/__redacted__ https://github.com/__redacted__
      |           |-git ls-remote https://github.com/__redacted__
      |           |   `-git-remote-http https://github.com/__redacted__ https://github.com/__redacted__
      |           |-git ls-remote https://github.com/__redacted__
      |           |   `-git-remote-http https://github.com/__redacted__ https://github.com/__redacted__
      |           |-git ls-remote https://github.com/__redacted__
      |           |   `-git-remote-http https://github.com/__redacted__ https://github.com/__redacted__
      |           |-git ls-remote https://github.com/__redacted__
      |           |   `-git-remote-http https://github.com/__redacted__ https://github.com/__redacted__
      |           |-git ls-remote https://github.com/__redacted__
      |           |   `-git-remote-http https://github.com/__redacted__ https://github.com/__redacted__
      |           |-git ls-remote https://github.com/__redacted__
      |           |   `-git-remote-http https://github.com/__redacted__ https://github.com/__redacted__
      |           |-git ls-remote https://github.com/__redacted__
      |           |   `-git-remote-http https://github.com/__redacted__ https://github.com/__redacted__
      |           |-git ls-remote https://github.com/__redacted__
      |           |   `-git-remote-http https://github.com/__redacted__ https://github.com/__redacted__
      |           |-git ls-remote https://github.com/__redacted__
      |           |   `-git-remote-http https://github.com/__redacted__ https://github.com/__redacted__
      |           |-git ls-remote https://github.com/__redacted__
      |           |-git ls-remote https://github.com/__redacted__
      |           |   `-git-remote-http https://github.com/__redacted__ https://github.com/__redacted__
      |           |-git ls-remote https://github.com/__redacted__
      |           |   `-git-remote-http https://github.com/__redacted__ https://github.com/__redacted__
      |           |-git ls-remote https://github.com/__redacted__
      |           |   `-git-remote-http https://github.com/__redacted__ https://github.com/__redacted__
      |           |-git ls-remote https://github.com/__redacted__
      |           |   `-git-remote-http https://github.com/__redacted__ https://github.com/__redacted__
      |           |-git ls-remote https://go.googlesource.com/sync
      |           |   `-git-remote-http https://go.googlesource.com/sync https://go.googlesource.com/sync
      |           |-git ls-remote https://go.googlesource.com/image
      |           |   `-git-remote-http https://go.googlesource.com/image https://go.googlesource.com/image
      |           |-git ls-remote https://github.com/__redacted__
      |           |   `-git-remote-http https://github.com/__redacted__ https://github.com/__redacted__
      |           |-git ls-remote https://github.com/__redacted__
      |           |   `-git-remote-http https://github.com/__redacted__ https://github.com/__redacted__
      |           |-git ls-remote https://go.googlesource.com/sys
      |           |-hg /usr/bin/hg paths
      |           `-106*[{dep}]

My ask is for a switch to limit the parallelism, or a way to disable it entirely.

[jmank88]

#431 will improve the situation for subsequent runs, but there is still value in some sort of external processes limit. I will look into how the mechanism in code might work. My gut is to use some sort of environment var, e.g. DEP_MAX_EXT_PROCS, but we can bike shed on the name.

[jmank88]

@sdboyer I'm exploring attaching a simple channel semaphore to context.Context (for gps.cmds to acquire inside CombinedOutput()), which requires more context plumbing, so going this route might resolve #423 as well.

[danielbprice]

DEP_MAX_EXT_PROCS (DEP_MAX_SUBPROCS?) would be exceedingly helpful. Thanks. I experimented with adding a semaphore myself but the code has enough layers that I couldn't quite figure out where to place it.

[jmank88]

I should have a PR up tonight, but plumbing context.Context around touched 61 files, and there are a few small things to sort out, so I can't say how soon it will merge.

If you need a quick-fix until my solution is merged, nearly everything goes through gps.cmd (all the get/fetch/update calls at least), so you could acquire a global semaphore from cmd.CombinedOutput().

[jmank88]

PR is up, if you want to test the branch.

[danielbprice]

@jmank88 I had the chance to test this branch on x86 and ARM (Raspberry Pi) tonight, and it worked as expected. I monitored with pstree and saw just as expected, a much more well controlled build. I also tried DEPMAXSUBPROCS=0, =1, =-1, =aa, etc. to be sure the edge cases work as expected.

Thanks a bunch for your hard work. I read/skimmed the patch and it's pretty enormous.

[mvdan]

Dep was officially deprecated earlier this year, and the proposal to archive this repository was accepted. As such, I'm closing outstanding issues before archiving the repository. For any further comments, please use the proposal thread on the Go issue tracker. Thanks!