Skip test_disk_offload for WhisperModelTest

[ydshieh]

What does this PR do?

Since #22486, WhisperModelTest.test_disk_offload to fail. I just blindly skip this test and guess it is ok.....?

[sgugger]

This might be worth investigating more, e.g. is it working for big models (cc @ArthurZucker ).

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[ydshieh]

I am not able to use openai/whisper-base as I get GPU OOM. When I tried to use openai/whisper-tiny.en, I have a hard time to change the parameters to get a working input dict for the model.

If I just changed the values in model tester to get a larger model (but random), I get different kinds of error like IndexError: list index out of range in dispatch_model or RuntimeError: Tensor on device meta is not on the expected device cuda:0! in model forward.

But even if I revert the change in #22486, the above attempts to use larger (fake) models still have the same issue. I guess we will have to look into this.

[ydshieh]

Convert to draft to avoid being merged.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.