Execution Log Sorting Causes Huge Memory Consumption when --execution_log_binary_file and --execution_log_json_file are enabled

[auzhva]

Description of the bug:

Both --execution_log_binary_file and --execution_log_json_file do execution log sorting. This requires Bazel to load the entire log into memory and if execution is big enough then it causes out of memory issues.

On real-life scenarios it's required to give up to --host_jvm_args=-Xmx64g just to let it sort a log.

While sorting may be useful for some scenarios, this is far from being necessary everywhere. For example, in our scenario execution log is needed only to understand how many non-cached actions were there and what are they. It's absolutely meaningless to spend all that memory to sort it.

I believe that sorting should be part of execution log comparison tools, not Bazel itself. Or at least Bazel should provide a way to opt-out from sorting for those for whom that is not needed.

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

1. Run bazel build --execution_log_binary_file=log.bin //some/big/target
2. See how it fails

Which operating system are you running Bazel on?

linux

What is the output of bazel info release?

release 6.0.0

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

Private repository

Have you found anything relevant by searching the web?

https://bazelbuild.slack.com/archives/CA31HN1T3/p1672037815196099

Any other information, logs, or outputs that you want to share?

No response

[sgowroji]

Hi @auzhva, Could you please provide an example to reproduce the above scenario. Thanks !

[auzhva]

Hi,

It requires something big enough. So some "minimal sample" wouldn't trigger the memory issue. Needs something like 50k actions at least, so the execution log would be big enough (few Gbs at least), so when it will load it for sorting it would run out of memory.

Especially visible on builds with 100k+ actions.

I'll try to fund something open source of that size and/or generate some synthetic tree with no-op rule maybe if necessary... So far at the moment couldn't share corporate repo of that size.

[tjgq]

I can totally believe that one might run out of memory when sorting very large execution logs. I'd be willing to review a PR adding a flag to disable sorting, if someone were to send it.

[YuanHao97]

@tjgq I opened a PR about this issue, could you help me review it?