Update cudnn from v8 to v9 across CUDA versions and x86/arm

[nWEIdia]

Re-land #1822

Supporting pytorch/pytorch#123475

Reference PR: #1271

cc @eqy @tinglvv @ptrblck @atalman @malfet

[tinglvv]

Thanks for preparing this! Let me test locally if the upgrade would break anything for ARM.

[tinglvv]

Suggested one change.
Built wheel with cudnnv9 and running into this error when running the tests:

Unable to load any of {libcudnn_engines_precompiled.so.9.1.0, libcudnn_engines_precompiled.so.9.1, libcudnn_engines_precompiled.so.9, libcudnn_engines_precompiled.so}
 File "/test-arm/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 453, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: CUDNN_BACKEND_TENSOR_DESCRIPTOR cudnnFinalize failed cudnn_status: CUDNN_STATUS_NOT_INITIALIZED`

@nWEIdia We will need to resolve this before merging the change.

[nWEIdia]

Suggested one change. Built wheel with cudnnv9 and running into this error when running the tests:

Unable to load any of {libcudnn_engines_precompiled.so.9.1.0, libcudnn_engines_precompiled.so.9.1, libcudnn_engines_precompiled.so.9, libcudnn_engines_precompiled.so}
 File "/test-arm/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 453, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: CUDNN_BACKEND_TENSOR_DESCRIPTOR cudnnFinalize failed cudnn_status: CUDNN_STATUS_NOT_INITIALIZED`

@nWEIdia We will need to resolve this before merging the change.

Thanks @tinglvv ! Could you please briefly describe the reproducer steps? I do know currently it has not been successful in building v9 based wheel, did you manually build a v9 based arm cuda wheel?

[bryantbiggs]

out of curiosity, do we plan to carry both the legacy API and new API here so that we can migrate pytorch over to the new graph API? this seems like the size impact would roughly double which as things currently stand ... its already quite a lot in terms of overall final artifact size

ref https://docs.nvidia.com/deeplearning/cudnn/latest/api/overview.html

[nWEIdia]

Good point and I think this question applies to x86 as well.
This PR was created not considering the size impact and ship both legacy and new API .so files, for both x86 and arm.

cc @ptrblck @eqy @malfet @atalman @tinglvv for additional inputs

[tinglvv]

Suggested one change. Built wheel with cudnnv9 and running into this error when running the tests:

Unable to load any of {libcudnn_engines_precompiled.so.9.1.0, libcudnn_engines_precompiled.so.9.1, libcudnn_engines_precompiled.so.9, libcudnn_engines_precompiled.so}
 File "/test-arm/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 453, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: CUDNN_BACKEND_TENSOR_DESCRIPTOR cudnnFinalize failed cudnn_status: CUDNN_STATUS_NOT_INITIALIZED`

@nWEIdia We will need to resolve this before merging the change.

Thanks @tinglvv ! Could you please briefly describe the reproducer steps? I do know currently it has not been successful in building v9 based wheel, did you manually build a v9 based arm cuda wheel?

Yes the wheel is built ok, just need to edit the cmake rules that @eqy mentioned in slack so that it recognizes cudnnv9 I think.

The required cmake changes seem to be from pytorch/pytorch#123475, which is failing cuda-aarch64 (due to this change #1847 missing). Due to this inter-dependency, it might be okay to ignore the cuda-aarch64 failures and merge the pytorch/pytorch change, then merge this change to fix the cuda-aarch64 failure.

Reproducer steps:

Step 1 Build image: GPU_ARCH_TYPE=cuda-aarch64 GPU_ARCH_VERSION=12.4 manywheel/build_docker.sh
Step 2 Run image: docker run --gpus all -it pytorch/manylinuxaarch64-builder:(replace with the generated docker from step 1)
Step 3 Git clone the pytorch repo to the docker image: cd / && git clone https://github.com/pytorch/pytorch.git
Step 4 Build the wheel: cd /builder/aarch64_linux && DESIRED_PYTHON=3.10 DESIRED_CUDA=12.4 ./aarch64_ci_build.sh