cmd/go: concurrent build and cache clean is unsafe

[josharian]

Migrated from discussion in #31931.

If you start a build with go build, and run go cache -clean while it is building, the build can fail, because intermediate artifacts get deleted.

A sample failure looks like:

# os/exec
../os/exec/exec.go:24:2: can't open import: "bytes": open /Users/josh/Library/Caches/go-build/85/852a5f87d98090fa893dcf8b168751b8a141cb0f4087b4951cfd9fc5edebe252-d: no such file or directory

@jayconrod wrote:

What should go clean -cache do with concurrent builds?

• We could have a "clean" lock that would block go build and other commands until go clean -cache finishes.
• go clean -cache could leave files created after it started. This might be hard to do reliably and portably though.

@bcmills wrote:

I think that locking scheme is possible to implement using the existing cmd/go/internal/lockedfile API.

go build would obtain a read-lock on the file (os.O_CREATE|os.O_RDONLY)
go clean would obtain a write-lock on the file (os.O_CREATE|os.O_WRONLY)

We don't currently rely on file-locking for the cache, but since this would only prevent a build / clean race perhaps it's not so bad.

I am not sure that @jayconrod's second option would work: I believe the problem is that the clean deletes files previously created by this build, which the build is still relying on.

@jayconrod's first option would work as long as go clean -cache is also blocked until go build is done. This makes it look a lot like a RW lock, as @bcmills notes. The only remaining danger is starvation of the clean. (That could actually happen in my particular, unusual case, since I am running many concurrent builds, but that's probably not worth designing around.)

[bcmills]

The only remaining danger is starvation of the clean.

We could address that by adding a second (meta) lock file, serializing acquisition of the clean lock.

Then go build would:

1. Acquire an exclusive lock on the meta file.
2. Acquire a shared lock on the clean file.
3. Release the meta lock.
4. Perform the build.
5. Release the clean lock.

go clean would:

1. Acquire an exclusive lock on the meta file.
2. Acquire an exclusive lock on the clean file.
3. Release the meta lock.
4. Clean the cache.
5. Release the clean lock.

That would force a total order on the start of commands, but allow the build operations to proceed in parallel once started.