Speed up tests by shaving off subprocess when not needed

[muellerzr]

What does this PR do?

As title states, subprocess is slow. Each time we call it adds a whole second or so to the testing suite. This PR removes that (when save/applicable) from our testing suite, and instead does a 1:1 replacement by capturing the stdout instead of a func (said func being the equivalent accelerate.commands.X.

Total CPU CI time:

Before: 2m53s
After: 2m36s

For specific areas:

• TpuConfigTester: (10x reduction)

Before: 10.77s
After: 0.95s

• AccelerateLauncherTester (50% reduction)

Before: 13.46s
After: 6.99s

Fixes # (issue)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@BenjaminBossan @SunMarc

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[BenjaminBossan]

Honestly, I'm a bit confused by the command launchers. It feels that there are multiple ways to achieve the same, or very similar things, and can't wrap my heard around what the difference is.

E.g. let's take the TPU tests. Previously, they were calling run_command, which in turn calls subprocess.check_output. Now they call tpu_command_launcher which calls subprocess.run. So how come the latter is faster? Is it really related to spawning subprocesses, as it appears that both ways do that.

Maybe you plan on harmonizing this further later on (say, removing run_command completely) and then this will become more obvious to me, but as is I'm confused.

That said, don't let my confusion block you from proceeding if this is a clear gain in test speed.

[muellerzr]

@BenjaminBossan yep exactly as it sounds, doing 2x subprocess is slower than 1x by a decent margin apparently. (Yes, I more or less intend to remove run_command, but not execute_subprocess_async as that's still needed I'm pretty sure)

[BenjaminBossan]

yep exactly as it sounds, doing 2x subprocess is slower than 1x by a decent margin apparently

Hmm, you mean that run_command runs subprocess twice? How is that? Also, is it not surprising that this would lead to a 10x reduction?

[muellerzr]

Only when the underlying command calls subprocess (like the tpu ones)

[BenjaminBossan]

I'm still not 100% sure why we see the speedup, but it's fine to merge from my point of view.