Upload large files directly from disk, without loading into memory

[huonw]

Is your feature request related to a problem? Please describe.

With #18153, large files produced are now cached as standalone files, on disk, rather than in the core LMDB structures. Uploading to a remote cache should be able to pipe these directly from that location on disk.

Currently they're pulled into memory, written to a temporary file (with an additional bug: it's sync IO in async code), and then that temporary file is uploaded via mmap. Breadcrumbs, in src/rust/engine/fs/store:

1. store::Store::ensure_remote_has_recursive calls store::Store::store_large_blob_remote
2. store_large_blob_remote calls store::remote::ByteStore::store_buffered
3. store_buffered creates temporary files...
4. ...then calls the closure that store_large_blob_remote provided with a std::fs::File for that temporary file
5. that closure then reads the whole large-file into memory and splats it into the temporary file (synchronously)

Describe the solution you'd like

store::Store::ensure_remote_has_recursive should pass a file handle into the remote cache, and that's manipulated/uploaded directly, without going through memory.

The following code could be adjusted to just decide whether the blob is on disk (if so, go via the disk) or is in LMDB (if so, just pull into memory), resolving the TODO by removing the linking to wire chunk size:

pants/src/rust/engine/fs/store/src/lib.rs

Lines 831 to 839 in 4c708a0

	// TODO(John Sirois): Consider allowing configuration of when to buffer large blobs
	// to disk to be independent of the remote store wire chunk size.
	if digest.size_bytes > remote_store.store.chunk_size_bytes() {
	Self::store_large_blob_remote(local, remote_store.store, entry_type, digest)
	.await
	} else {
	Self::store_small_blob_remote(local, remote_store.store, entry_type, digest)
	.await
	}

Describe alternatives you've considered

N/A

Additional context
Add any other context or screenshots about the feature request here.

[huonw]

Hmmm, I wonder if the use of an mmap'd file in async code is also an async hazard: I'm imagining memory accesses to pages of the file that aren't yet in memory will trigger effectively a sync read (i.e. the page fault will deschedule the whole OS thread including the Rust-level async runtime, rather than letting the runtime work on another coroutine...).