the dynamic spawn strategy is broken for remote execution

[buchgr]

We have received several bug reports about the dynamic spawn strategy not working properly with remote execution / caching. I took some time to look through the code to better understand the issue. I believe to now understand the heart of the issue.

The dynamic spawn strategy in a single line:

SpawnResult result =
ExecutorService.invokeAny(localExecutionCallable, remoteExecutionCallable)`

This line takes the result of the first callable to finish and calls Future.cancel(mayInterruptIfRunning=true) on the slower one. The bug is that the Callable's have side effects in that both write their outputs to the file system and incidentally also to the same files.

This has two issues:

1. The effective output files of a single action can end up being a mix of both local and remote execution. This would mostly go unnoticed (esp. if the actions are reproducible) except when it doesn't then it would lead to hard to debug correctness bugs.
2. The output files Bazel sees can be corrupted. This can be the case if both the remote strategy and local execution strategy write to the same file concurrently.

I believe 2.) is what users are most likely to see.

[buchgr]

The fix is to call SpawnExecutionContext.lockOutputfiles() before starting downloading of outputs.