capturing/monitoring files from absolute paths?

[tdyas]

Go 1.20 will no longer ship with precompiled package archives for the standard library. See #17950. This means Pants will need to compile the Go standard library from source from filesystem paths.

Even if the Go backend compiles the stdlib from the sources in the filesystem (versus capturing to a digest), the Go backend will at least still need to monitor those paths for changes so it can properly invalidate cached results. (An alternative would be to capture Go stdlib sources to the Pants cache and compile "normally".)

1. What does the Pants engine currently support with respect to files at absolute paths?
2. Any advice on how to proceed here?

[benjyw]

Thanks for looking into this! Never a dull moment in Go.

ccing @thejcannon who recently did a bunch of work on sources from URLs. I wonder if we can leverage that somehow. Those required manually specifying a SHA, which would be onerous here, but maybe we can figure something out for local URLs.

[kaos]

Or perhaps support an auxiliary URL for fetching a checksums file--as there normally is one next to the binaries. Doesn't give the same level of tamper protection, but still protects against accidental corruption.

[thejcannon]

The engine supports file:// URLs already. Does that help?

[thejcannon]

Also @kaos I just introduced arbitrary URL handlers, so doing checksum verification in a plugin isn't far off

[benjyw]

I'm guessing this is a whole directory's worth of files, whose names are not known in advance?

[tdyas]

Correct. The files here are from the unpacked GOROOT. We won't know the files on disk in advance and only the packed archive will have had a checksum computed.

One issue here is invalidating properly if the path to the unpacked Go SDK remains the same but with the contents change, say for a new version unpacked to /usr/local/go. We currently rely on go version to do that. We could just continue to do that.

I ask the question about capture/monitoring absolute paths in case there is a more robust way to ensure we invalidate properly.

I should not be blocked if we are okay with compiling from the absolute paths and output/capture the sandbox for actual package archive with only version check.

[stuhood]

#10842 covers capturing snapshots from absolute paths. A little bit of work (see the cfg! blocks referring to macos) would be required to move our current file watching from "exactly one statically known root" to "multiple, possibly dynamically added roots".