Question: does ensure need to reach out to the network as much as it does?

[peterbourgon]

This is feedback based on my first run with the tool. I was surprised that ensure seemed to always re/query each dep's source of truth, and re/solve the entire graph, even if I was just updating a single dep. It led to significant frustration due to mismatched expectations. Is this necessary? Can it be improved with e.g. a local cache?

[sdboyer]

(Almost) all of this comes down to how gps operates.

I was surprised that ensure seemed to always re/query each dep's source of truth,

gps can't not do this at the moment, as it needs the list of possible versions for each project to operate on. (Though there is a happy path where, if we have data in the lock and are not upgrading, we avoid fetching versions until we know we have to break the lock). We could cache version lists locally, and that's a goal. However, at least part of the slowness issue with this was that git repo sources were blocking the whole solving algorithm in a way that they weren't supposed to. That should be fixed...though allowing more parallelization there might also have the effect of exacerbating #66.

and re/solve the entire graph, even if I was just updating a single dep. It led to significant frustration due to mismatched expectations. Is this necessary?

This one's just the nature of the beast. Any update can potentially introduce new transitive deps, including new shared deps, which may result in conflicts; it's unsafe to treat any update in isolation.

Can it be improved with e.g. a local cache?

Yes, enormously. Here's one of the solver wall time listings from your testing runs:

Solver wall times by segment:
     b-source-exists: 1m38.425418832s
              b-gmal: 1m25.467943447s
         b-list-pkgs:    288.814573ms
         select-atom:     15.498336ms
             satisfy:      2.574273ms
            new-atom:      1.566509ms
  b-deduce-proj-root:       679.679µs
         select-root:       552.279µs
               other:       106.195µs
     b-list-versions:        22.926µs

All three of the top items are completely cache-friendly. Dealing with each:

• The first, b-source-exists, is slow because it's the issue I mentioned above that I already fixed. Lots of wall time spent there because each git project was triggering a git ls-remote call that blocked solver progress. Right now, barring timeouts, I'd guess/hope that that time would be cut from 100s to 5-10s or so. That time will likely start showing up under b-list-versions instead, so referring to that next...
• b-list-versions is cacheable, with pretty typical caching tradeoffs: when we cache, we don't retrieve the most recent version list from upstream. The exact risks there vary a bit, but the bad outcomes are basically either:
  • A solve fails that would have succeeded if the most recent version information was used
  • A solve succeeds that would have failed, or found a different solution, if the most recent version was used. This could also be thought of as a "non-portable" solution.
• b-gmal ("Get Manifest And Lock") - this is pretty much a caching dream. The call ultimately invokes the DeriveManifestAndLock() method on our analyzer, which it's assumed will always return the same value as long as the code it operates on is the same. That means the result should be permanently cacheable when keyed by, say, git revision.
• b-list-pkgs: it's just parsing trees of go code to create gps.PackageTrees. Same caching properties as b-gmal.

It won't be totally smooth sailing, but gps.SourceMgr was mostly designed with the goal of dropping just such a caching layer into place. We could do it by writing a bunch of files out to disk for each discrete project, or by keeping a single db with all the info. I'm not sure which I favor.

Once that caching's in place - and especially if we put logic in the source manager to pre-warm these caches in the background of solve runs - then I expect we'll see solve times drop precipitously. select-atom and satisfy have some unoptimized paths, but even without that, non-pathological solving cases should generally complete in the millisecond range. (I hope 👼)

Is this necessary?

Returning to this just to say...no. I need to open issues for many of the above-named things, and this one, too: there needs to be an option that seals off gps from any network activity. It's an important operational mode to offer, just in general.

Once the caches are in place, it should be reasonably easy to accomplish this - it'll just operate from cache, or fail. The one thing that might be a bit hinky is dealing with imports that rely on go-get metadata to resolve. That can probably be stuck in the cache, too, of course, but...well, that's a new class of risk.

[sdboyer]

Closing this in favor of the more formal #289