Status of golang support? Seems very slow overall

[HenrikPoulsen]

I saw in the docs that golang support is in the beta phase, however it doesn't really address how usable it is.
I tried it out on a monorepo I have with 10 or so local modules (~80 dependencies and ~200 indirect dependencies, aka pretty small as monorepos go) and it seems very rough around the edges for sure.

The main issue/deal breaker imo is that in my case it takes 20+ minutes to just download and analyze and build all the modules.
It also ends up taking ~90% of the available system memory (of which I have 64GB total) while it's running

Why does it take this long and why does it not, seemingly, use the GOPATH cache for this which already has everything downloaded?
Is this expected and by design?
Is there a config issue on my end? If so how do you debug that?

By comparison go test ./... after doing a go clean --modcache it takes ~10 seconds to pull all the packages. On subsequent runs it takes no measurable time before it starts running the tests. For pantsbuild it takes 60+ seconds on a rerun before it starts running any tests, since it seems to be rechecking all the dependencies again, quite slowly.

Is this the expected behavior or is there some other issue at play here for my specific setup?

[benjyw]

Thanks for starting the discussion. Going to pull @tdyas in, as he's the Golang guru, and is better placed than I am to answer specific technical questions, especially around the performance questions, which do seem pathological, and may fixable. That said, slow "time to first test" is an ongoing issue with Pants, across all backends, and one we are looking at various ways of addressing.

FWIW I think the status of the golang backend is "needs ownership/sponsorship". Anyone can jump in and work on the parts of Pants that they care about, so where we see the most PRs is an indication of the priorities of the more active maintainers (who work on Pants on behalf of their companies). The golang backend sees relatively few such PRs, and is therefore maintained on a best-effort basis by the core maintainers, who likely aren't using it themselves for their own projects, and so aren't exposed to such issues on a personal and regular basis.

We would love to have more contributors to the golang backend. Or alternatively, it is possible to contract projects on it out to freelance members of the community. See https://www.pantsbuild.org/spotlight/service-providers for example.

[frekw]

We're also experiencing this.

After having exported the pants diagnostics and imported it as traces, it seems like the majority of the time is spent in

pants/src/python/pants/backend/go/target_type_rules.py

Line 361 in 6cac603

async def generate_targets_from_go_mod(

[benjyw]

@tdyas this tracks with what we know, right?