0.26.0rc8 crash with experimental_execution_log_file

[keith]

When building with 0.26.0rc8 and passing experimental_execution_log_file, bazel crashes with:

Internal error thrown during build. Printing stack trace: java.lang.IllegalArgumentException
        at com.google.common.base.Preconditions.checkArgument(Preconditions.java:127)
        at com.google.devtools.build.lib.vfs.FileSystem.getPath(FileSystem.java:78)
        at com.google.devtools.build.lib.bazel.SpawnLogModule.initOutputs(SpawnLogModule.java:94)
        at com.google.devtools.build.lib.bazel.SpawnLogModule.executorInit(SpawnLogModule.java:119)
        at com.google.devtools.build.lib.buildtool.ExecutionTool.<init>(ExecutionTool.java:137)
        at com.google.devtools.build.lib.buildtool.BuildTool.buildTargets(BuildTool.java:151)
        at com.google.devtools.build.lib.buildtool.BuildTool.processRequest(BuildTool.java:272)
        at com.google.devtools.build.lib.runtime.commands.BuildCommand.exec(BuildCommand.java:97)
        at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.execExclusively(BlazeCommandDispatcher.java:498)
        at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.exec(BlazeCommandDispatcher.java:206)
        at com.google.devtools.build.lib.server.GrpcServerImpl.executeCommand(GrpcServerImpl.java:749)
        at com.google.devtools.build.lib.server.GrpcServerImpl.access$1600(GrpcServerImpl.java:103)
        at com.google.devtools.build.lib.server.GrpcServerImpl$2.lambda$run$0(GrpcServerImpl.java:818)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)

INFO: Elapsed time: 4.007s
INFO: 0 processes.
FAILED: Build did NOT complete successfully
Internal error thrown during build. Printing stack trace: java.lang.IllegalArgumentException
        at com.google.common.base.Preconditions.checkArgument(Preconditions.java:127)
        at com.google.devtools.build.lib.vfs.FileSystem.getPath(FileSystem.java:78)
        at com.google.devtools.build.lib.bazel.SpawnLogModule.initOutputs(SpawnLogModule.java:94)
        at com.google.devtools.build.lib.bazel.SpawnLogModule.executorInit(SpawnLogModule.java:119)
        at com.google.devtools.build.lib.buildtool.ExecutionTool.<init>(ExecutionTool.java:137)
        at com.google.devtools.build.lib.buildtool.BuildTool.buildTargets(BuildTool.java:151)
        at com.google.devtools.build.lib.buildtool.BuildTool.processRequest(BuildTool.java:272)
        at com.google.devtools.build.lib.runtime.commands.BuildCommand.exec(BuildCommand.java:97)
        at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.execExclusively(BlazeCommandDispatcher.java:498)
        at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.exec(BlazeCommandDispatcher.java:206)
        at com.google.devtools.build.lib.server.GrpcServerImpl.executeCommand(GrpcServerImpl.java:749)
        at com.google.devtools.build.lib.server.GrpcServerImpl.access$1600(GrpcServerImpl.java:103)
        at com.google.devtools.build.lib.server.GrpcServerImpl$2.lambda$run$0(GrpcServerImpl.java:818)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
java.lang.IllegalArgumentException
        at com.google.common.base.Preconditions.checkArgument(Preconditions.java:127)
        at com.google.devtools.build.lib.vfs.FileSystem.getPath(FileSystem.java:78)
        at com.google.devtools.build.lib.bazel.SpawnLogModule.initOutputs(SpawnLogModule.java:94)
        at com.google.devtools.build.lib.bazel.SpawnLogModule.executorInit(SpawnLogModule.java:119)
        at com.google.devtools.build.lib.buildtool.ExecutionTool.<init>(ExecutionTool.java:137)
        at com.google.devtools.build.lib.buildtool.BuildTool.buildTargets(BuildTool.java:151)
        at com.google.devtools.build.lib.buildtool.BuildTool.processRequest(BuildTool.java:272)
        at com.google.devtools.build.lib.runtime.commands.BuildCommand.exec(BuildCommand.java:97)
        at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.execExclusively(BlazeCommandDispatcher.java:498)
        at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.exec(BlazeCommandDispatcher.java:206)
        at com.google.devtools.build.lib.server.GrpcServerImpl.executeCommand(GrpcServerImpl.java:749)
        at com.google.devtools.build.lib.server.GrpcServerImpl.access$1600(GrpcServerImpl.java:103)
        at com.google.devtools.build.lib.server.GrpcServerImpl$2.lambda$run$0(GrpcServerImpl.java:818)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
java.lang.IllegalArgumentException
        at com.google.common.base.Preconditions.checkArgument(Preconditions.java:127)
        at com.google.devtools.build.lib.vfs.FileSystem.getPath(FileSystem.java:78)
        at com.google.devtools.build.lib.bazel.SpawnLogModule.initOutputs(SpawnLogModule.java:94)
        at com.google.devtools.build.lib.bazel.SpawnLogModule.executorInit(SpawnLogModule.java:119)
        at com.google.devtools.build.lib.buildtool.ExecutionTool.<init>(ExecutionTool.java:137)
        at com.google.devtools.build.lib.buildtool.BuildTool.buildTargets(BuildTool.java:151)
        at com.google.devtools.build.lib.buildtool.BuildTool.processRequest(BuildTool.java:272)
        at com.google.devtools.build.lib.runtime.commands.BuildCommand.exec(BuildCommand.java:97)
        at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.execExclusively(BlazeCommandDispatcher.java:498)
        at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.exec(BlazeCommandDispatcher.java:206)
        at com.google.devtools.build.lib.server.GrpcServerImpl.executeCommand(GrpcServerImpl.java:749)
        at com.google.devtools.build.lib.server.GrpcServerImpl.access$1600(GrpcServerImpl.java:103)
        at com.google.devtools.build.lib.server.GrpcServerImpl$2.lambda$run$0(GrpcServerImpl.java:818)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
FAILED: Build did NOT complete successfully

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

bazel-bin/src/bazel build src:bazel --experimental_execution_log_file=foo.log

What operating system are you running Bazel on?

Ubuntu 16.04 or macOS

What's the output of bazel info release?

release 0.26.0rc8

Have you found anything relevant by searching the web?

• Maybe this flag is being replaced? Document --execution_log_json_file  #8359

[laurentlb]

cc @agoulti, @buchgr

Can you look at this?
Should it be replaced with execution_log_json_file?

(General comment regarding the release policy: experimental flags are normally not supported and shouldn't qualify for cherry-picks. However, I'm not familiar with this particular case, I'll let other people comment on it)

[keith]

I'm happy to replace our case with execution_log_json_file if that's the solution. With rc8 that doesn't crash

[keith]

With this new flag I'm still seeing issues on macOS and ubuntu:

WARNING: Execution log might not have been populated. Raw execution log is at /tmp/exec5665122725438342177.log
Internal error thrown during build. Printing stack trace: java.lang.IllegalArgumentException
	at com.google.common.base.Preconditions.checkArgument(Preconditions.java:127)
	at com.google.devtools.build.lib.bazel.execlog.StableSort.lambda$stableSort$0(StableSort.java:100)
	at java.base/java.util.Comparator.lambda$comparing$77a9974f$1(Unknown Source)
	at java.base/java.util.PriorityQueue.siftUpUsingComparator(Unknown Source)
	at java.base/java.util.PriorityQueue.siftUp(Unknown Source)
	at java.base/java.util.PriorityQueue.offer(Unknown Source)
	at java.base/java.util.PriorityQueue.add(Unknown Source)
	at com.google.devtools.build.lib.bazel.execlog.StableSort.stableSort(StableSort.java:116)
	at com.google.devtools.build.lib.bazel.execlog.StableSort.stableSort(StableSort.java:60)
	at com.google.devtools.build.lib.bazel.SpawnLogModule.afterCommand(SpawnLogModule.java:142)
	at com.google.devtools.build.lib.runtime.BlazeRuntime.afterCommand(BlazeRuntime.java:580)
	at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.execExclusively(BlazeCommandDispatcher.java:506)
	at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.exec(BlazeCommandDispatcher.java:206)
	at com.google.devtools.build.lib.server.GrpcServerImpl.executeCommand(GrpcServerImpl.java:749)
	at com.google.devtools.build.lib.server.GrpcServerImpl.access$1600(GrpcServerImpl.java:103)
	at com.google.devtools.build.lib.server.GrpcServerImpl$2.lambda$run$0(GrpcServerImpl.java:818)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
java.lang.IllegalArgumentException

[agoulti]

So the problem with --experimental_execution_log_file manifests itself when the relative path is used.
I will submit a fix shortly. Absolute path should work.

The problem with execution_log_json_file is a violated assumption: the stableSort algorithm assumes that all the actions have outputs, and the log violates that assumption. Would you be willing to share what kind of action produced this error? (in the execution log it would be an action with an empty set of actual output files)

[buchgr]

the stableSort algorithm assumes that all the actions have outputs, and the log violates that assumption.

that's generally not true ... sounds like we should patch the algorithm to no longer assume this. should be easy enough?

[keith]

Here's a "raw" log from the command before bazel crashes: foo.log

It still seems to be parseable with execlog:parser, let me know if you want something else