Fix TPU testing and collect all tests

[awaelchli]

What does this PR do?

Fixes #13720

Addresses a range of issues with the TPU CI:

• Now collects all tests that are marked with RunIf(tpu=True). No longer do the tests get hardcoded and forgotten, leading to tests being added that never run.
• Removed the @pl_multi_process decorator on all tests. This decorator suppressed exceptions and assertion errors. Some tests were broken and outdated for a while and were never raising the errors due to this.
• Added standalone marker RunIf(tpu=True, standalone=True) for TPU tests as an alternative to the aforementioned pl_multi_process. The CI now runs standalone tests similar to the GPU test suite. This is necessary for example when we need to run with the single device tpu strategy that requires us to access the xla device in the main process.
• Fixed tests that were affected by the aforementioned cause for failures.

Note, after resolving the core issues, and pushing many commits to run the CI, I have not seen any flakiness and random behavior anymore.

In a follow-up, we should update the CI to the latest torch_xla version.

Before submitting

• Was this discussed/approved via a GitHub issue? (not for typos and docs)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure your PR does only one thing, instead of bundling different changes together?
• Did you make sure to update the documentation with your changes? (if necessary)
• Did you write any new necessary tests? (not for typos and docs)
• Did you verify new and existing tests pass locally with your changes?
• Did you update the CHANGELOG? (not for typos, docs, test updates, or internal minor changes/refactorings)

PR review

Anyone in the community is free to review the PR once the tests have passed.
Before you start reviewing make sure you have read Review guidelines. In short, see the following bullet-list:

• Is this pull request ready for review? (if not, please submit in draft mode)
• Check that all items from Before submitting are resolved
• Make sure the title is self-explanatory and the description concisely explains the PR
• Add labels and milestones (and optionally projects) to the PR so it can be classified

Did you have fun?

I made sure I had fun coding 🙃

Part of #1 (it's a lie, this is just here to avoid noisy GitHub bot)

cc @Borda @tchaton @rohitgr7 @akihironitta @carmocca @kaushikb11

[Borda]

I think that in past we had just a selection of files to have it faster, but I agree that this is a more robust solution - we test all PL and we do not need to remember to add a single file somewhere...

[awaelchli]

TPU ci is still getting stuck/ timing out. One of the newly discovered tests must be the cause of this.

[tchaton]

TPU ci is still getting stuck/ timing out. One of the newly discovered tests must be the cause of this.

Any progress on this? Did you identify the hanging test?

[kaushikb11]

Any progress on this? Did you identify the hanging test?

I will take a stab at it today.