CUDA Capability Issue with "current" PyTorch

[amirhusseinab]

Question/Support Request

Running FastSurfer on our HPC cluster with Nvidia H100 fails with an error shown below. Any comments are appreciated.

Screenshots / Log files

Illegal option --
/venv/lib/python3.10/site-packages/torch/cuda/__init__.py:173: UserWarning: 
NVIDIA H100 with CUDA capability sm_90 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70 sm_75 sm_80 sm_86.
If you want to use the NVIDIA H100 GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))
Traceback (most recent call last):
  File "/fastsurfer/FastSurferCNN/run_prediction.py", line 630, in <module>
    raise e
  File "/fastsurfer/FastSurferCNN/run_prediction.py", line 566, in <module>
    pred_data = eval.get_prediction(
  File "/fastsurfer/FastSurferCNN/run_prediction.py", line 377, in get_prediction
    pred_prob = torch.zeros(shape, **kwargs)
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Environment

• FastSurfer Version: fastsurfer_2.2.0

absl-py                 2.0.0        /venv/lib/python3.10/site-packages conda
aiohttp                 3.9.1        /venv/lib/python3.10/site-packages conda
aiosignal               1.3.1        /venv/lib/python3.10/site-packages conda
async-timeout           4.0.3        /venv/lib/python3.10/site-packages conda
attrs                   23.1.0       /venv/lib/python3.10/site-packages conda
blinker                 1.7.0        /venv/lib/python3.10/site-packages conda
Brotli                  1.1.0        /venv/lib/python3.10/site-packages conda
cached-property         1.5.2        /venv/lib/python3.10/site-packages conda
cachetools              5.3.2        /venv/lib/python3.10/site-packages conda
certifi                 2023.11.17   /venv/lib/python3.10/site-packages conda
cffi                    1.16.0       /venv/lib/python3.10/site-packages conda
charset-normalizer      3.3.2        /venv/lib/python3.10/site-packages conda
click                   8.1.7        /venv/lib/python3.10/site-packages conda
cloudpickle             3.0.0        /venv/lib/python3.10/site-packages conda
cmake                   3.28.0       /venv/lib/python3.10/site-packages pip
colorama                0.4.6        /venv/lib/python3.10/site-packages conda
contourpy               1.2.0        /venv/lib/python3.10/site-packages conda
cryptography            41.0.7       /venv/lib/python3.10/site-packages conda
cycler                  0.12.1       /venv/lib/python3.10/site-packages conda
cytoolz                 0.12.2       /venv/lib/python3.10/site-packages conda
dask                    2023.12.0    /venv/lib/python3.10/site-packages conda
Deprecated              1.2.14       /venv/lib/python3.10/site-packages pip
filelock                3.13.1       /venv/lib/python3.10/site-packages pip
fonttools               4.46.0       /venv/lib/python3.10/site-packages conda
frozenlist              1.4.0        /venv/lib/python3.10/site-packages conda
fsspec                  2023.12.2    /venv/lib/python3.10/site-packages conda
google-auth             2.25.2       /venv/lib/python3.10/site-packages conda
google-auth-oauthlib    1.0.0        /venv/lib/python3.10/site-packages conda
grpcio                  1.60.0       /venv/lib/python3.10/site-packages conda
h5py                    3.7.0        /venv/lib/python3.10/site-packages conda
humanize                4.9.0        /venv/lib/python3.10/site-packages pip
idna                    3.6          /venv/lib/python3.10/site-packages conda
imagecodecs             2023.9.18    /venv/lib/python3.10/site-packages conda
imageio                 2.31.5       /venv/lib/python3.10/site-packages conda
importlib-metadata      7.0.0        /venv/lib/python3.10/site-packages conda
Jinja2                  3.1.2        /venv/lib/python3.10/site-packages pip
joblib                  1.3.2        /venv/lib/python3.10/site-packages conda
kiwisolver              1.4.5        /venv/lib/python3.10/site-packages conda
lapy                    1.0.1        /venv/lib/python3.10/site-packages conda
lit                     17.0.6       /venv/lib/python3.10/site-packages pip
locket                  1.0.0        /venv/lib/python3.10/site-packages conda
Markdown                3.5.1        /venv/lib/python3.10/site-packages conda
MarkupSafe              2.1.3        /venv/lib/python3.10/site-packages conda
matplotlib              3.7.1        /venv/lib/python3.10/site-packages conda
mpmath                  1.3.0        /venv/lib/python3.10/site-packages pip
multidict               6.0.4        /venv/lib/python3.10/site-packages conda
munkres                 1.1.4        /venv/lib/python3.10/site-packages
networkx                3.2.1        /venv/lib/python3.10/site-packages conda
nibabel                 5.1.0        /venv/lib/python3.10/site-packages conda
numpy                   1.25.0       /venv/lib/python3.10/site-packages conda
oauthlib                3.2.2        /venv/lib/python3.10/site-packages conda
packaging               23.2         /venv/lib/python3.10/site-packages conda
pandas                  1.5.3        /venv/lib/python3.10/site-packages
partd                   1.4.1        /venv/lib/python3.10/site-packages conda
Pillow                  10.0.1       /venv/lib/python3.10/site-packages conda
pip                     23.3.1       /venv/lib/python3.10/site-packages
platformdirs            4.1.0        /venv/lib/python3.10/site-packages conda
plotly                  5.18.0       /venv/lib/python3.10/site-packages conda
ply                     3.11         /venv/lib/python3.10/site-packages
pooch                   1.8.0        /venv/lib/python3.10/site-packages conda
protobuf                4.24.4       /venv/lib/python3.10/site-packages conda
psutil                  5.9.5        /venv/lib/python3.10/site-packages conda
pyasn1                  0.5.1        /venv/lib/python3.10/site-packages conda
pyasn1-modules          0.3.0        /venv/lib/python3.10/site-packages conda
pycparser               2.21         /venv/lib/python3.10/site-packages conda
PyJWT                   2.8.0        /venv/lib/python3.10/site-packages conda
pyOpenSSL               23.3.0       /venv/lib/python3.10/site-packages conda
pyparsing               3.1.1        /venv/lib/python3.10/site-packages conda
PyQt5                   5.15.9       /venv/lib/python3.10/site-packages conda
PyQt5-sip               12.12.2      /venv/lib/python3.10/site-packages
PySocks                 1.7.1        /venv/lib/python3.10/site-packages conda
python-dateutil         2.8.2        /venv/lib/python3.10/site-packages conda
pytz                    2023.3.post1 /venv/lib/python3.10/site-packages conda
pyu2f                   0.1.5        /venv/lib/python3.10/site-packages conda
PyWavelets              1.4.1        /venv/lib/python3.10/site-packages conda
PyYAML                  6.0.1        /venv/lib/python3.10/site-packages conda
requests                2.31.0       /venv/lib/python3.10/site-packages conda
requests-oauthlib       1.3.1        /venv/lib/python3.10/site-packages conda
rsa                     4.9          /venv/lib/python3.10/site-packages conda
scikit-image            0.19.3       /venv/lib/python3.10/site-packages conda
scikit-learn            1.2.2        /venv/lib/python3.10/site-packages conda
scikit-sparse           0.4.12       /venv/lib/python3.10/site-packages
scipy                   1.10.1       /venv/lib/python3.10/site-packages conda
setuptools              67.8.0       /venv/lib/python3.10/site-packages
SimpleITK               2.2.1        /venv/lib/python3.10/site-packages pip
sip                     6.7.12       /venv/lib/python3.10/site-packages conda
six                     1.16.0       /venv/lib/python3.10/site-packages conda
sympy                   1.12         /venv/lib/python3.10/site-packages pip
tenacity                8.2.3        /venv/lib/python3.10/site-packages conda
tensorboard             2.12.1       /venv/lib/python3.10/site-packages conda
tensorboard-data-server 0.7.0        /venv/lib/python3.10/site-packages conda
tensorboard-plugin-wit  1.8.1        /venv/lib/python3.10/site-packages conda
threadpoolctl           3.2.0        /venv/lib/python3.10/site-packages conda
tifffile                2023.12.9    /venv/lib/python3.10/site-packages conda
toml                    0.10.2       /venv/lib/python3.10/site-packages conda
tomli                   2.0.1        /venv/lib/python3.10/site-packages conda
toolz                   0.12.0       /venv/lib/python3.10/site-packages conda
torch                   2.0.1+cu117  /venv/lib/python3.10/site-packages pip
torchio                 0.18.83      /venv/lib/python3.10/site-packages pip
torchvision             0.15.2+cu117 /venv/lib/python3.10/site-packages pip
tornado                 6.3.3        /venv/lib/python3.10/site-packages conda
tqdm                    4.65.0       /venv/lib/python3.10/site-packages conda
triton                  2.0.0        /venv/lib/python3.10/site-packages pip
typing_extensions       4.9.0        /venv/lib/python3.10/site-packages conda
unicodedata2            15.1.0       /venv/lib/python3.10/site-packages conda
urllib3                 2.1.0        /venv/lib/python3.10/site-packages conda
Werkzeug                3.0.1        /venv/lib/python3.10/site-packages conda
wheel                   0.42.0       /venv/lib/python3.10/site-packages
wrapt                   1.16.0       /venv/lib/python3.10/site-packages pip
yacs                    0.1.8        /venv/lib/python3.10/site-packages conda
yarl                    1.9.3        /venv/lib/python3.10/site-packages conda
zipp                    3.17.0       /venv/lib/python3.10/site-packages conda

• Installation type: singularity
• OS: Linux
• GPU: Nvidia H100

[m-reuter]

This can probably only be solved by upgrading our torch dependency to cu118.

[amirhusseinab]

Thanks Martin. Then, I'll continue without implementing GPU then. Do you plan to upgrade the Torch dependency in future releases?

[m-reuter]

yes, actually we are finalising a new release, which we are currently delaying a little to add this and also make sure that native install works on the Mac. See our open PRs, specifically #560

[m-reuter]

Closing this for now, as it should be fixed in our next release and in dev (once the PR #560 is merged). Feel free to re-open if issues remain then.