[Release] Training operator 1.8.1 release

[tenzen-y]

This is the tracking issue for Training Operator v1.8.1 patch release for the Kubeflow 1.9.1 patch release.

Control Plane

• Broken preemption on TFJob with non default runPolicy.ttlSecondsAfterFinished #2239
• fix: suspend job status can not turn into Suspended and report error… #2159

SDK

• [SDK] Fix trainer error: Update the version of base image and add "num_labels" for downloading pretrained models #2230
• Update huggingface_hub Version in the storage initializer to fix ImportError #2180

cc @kubeflow/wg-training-leads

@andreyvelich Could you list SDK bug fixes?

[tenzen-y]

cc: @alculquicondor @mszadkow

[andreyvelich]

I think, this list looks good to me.
Let me cherry-pick SDK changes into release branch.

[alculquicondor]

Is there a target date?

[tenzen-y]

Is there a target date?

The kubeflow release team is planning the deadline now.

[tenzen-y]

Is there a target date?

The kubeflow release team is planning the deadline now.

@alculquicondor We quickly checked the deadline with the release team. In conclusion, we want to release the patch version by mid-September. So, I think it would be better to fix all the bugs by early September.

[tenzen-y]

All bugs were resolved.
@kubeflow/wg-training-leads If you have seen any other bugs, please let me know.

[andreyvelich]

@tenzen-y Please cherry-pick these PRs to the release branch.

[tenzen-y]

@tenzen-y Please cherry-pick these PRs to the release branch.

Sure.