benchmarks/torchbench_model: some benchmarks fail to load and kill experiment_runner's main process

[cota]

🐛 Bug

In dfcf306e7 Apply precision config env vars in the root process. (#6152)
we started running load_benchmark() from experiment_runner's
main process. Unfortunately, load_benchmark() for
some models does exit the calling process.
This results in experiment_runner exiting prematurely.

To Reproduce

Try to run under XLA any of the benchmarks added to the deny list in #6199. For example:

python xla/benchmarks/experiment_runner.py --dynamo=openxla --dynamo=openxla_eval --xla=PJRT --test=eval --test=train --accelerator=cuda --output-dirname=/tmp/pix2pix --repeat=5 --print-subprocess --suite-name=torchbench --filter='^pytorch_CycleGAN_and_pix2pix$' --log-level=debug ; echo $?

Note: pytorch_CycleGAN_and_pix2pix also fails early under inductor.

Expected behavior

The above should print a 0 exit code regardless of whether the benchmark fails to run or not. However, it prints 2.

Environment

• Reproducible on XLA backend [CPU/TPU]: GPU
• torch_xla version: dfcf306 and later.

[yeounoh]

@cota thanks for addressing this, can we close this issue now?

[cota]

We're working around this issue by temporarily disabling the affected benchmarks. But AFAICT this is still an issue. If you want we can close it -- I won't be working on this in the near future.
Maybe @ysiraichi will? Let's have him decide what to do with this issue.

[ysiraichi]

I'm only running dynamo+openxla tests, here. And, so far, I can't really reproduce these loading errors:

• pytorch_CycleGAN_and_pix2pix: runs eval and train successfully
• pytorch_unet: runs eval and train successfully if we don't throw on AMP
• tacotron2: eval and train fail at execution time (not when it's loading the model)