feat(ansible): upgrade for CUDA, TensorRT and CUDNN

[amadeuszsz]

Description

Copy of #5608 with fix #5730.

Reverted by #5729 due to incompatibility of upgraded dependencies with 0.40.0 autoware.universe release.

This PR can be opened after autoware 0.41.0 release.

How was this PR tested?

Notes for reviewers

None.

Effects on system behavior

None.

[github-actions]

Thank you for contributing to the Autoware project!

🚧 If your pull request is in progress, switch it to draft mode.

Please ensure:

• You've checked our contribution guidelines.
• Your PR follows our pull request guidelines.
• All required CI checks pass before marking the PR ready for review.

[xmfcx]

• I'll test https://autowarefoundation.github.io/autoware-documentation/main/tutorials/ad-hoc-simulation/rosbag-replay-simulation/
  • Without this PR.
  • With this PR.
• I'll use autoware-nightly.repos only.
• I will report my findings within this post.

Without the PR

Rosbag Replay Sim Demo

✅

AWSIM Labs Demo

✅

Had to merge these 2 PRs:

• fix(common_awsim_labs_sensor_launch): update the glog plugin namespace awsim_labs_sensor_kit_launch#22
• feat(awsim_labs_sensor_kit_launch): awsim lab load concatenate node parameters awsim_labs_sensor_kit_launch#9

With the PR

First removed the following files to regenerate:

~/autoware_data/traffic_light_classifier/ped_traffic_light_classifier_mobilenetv2_batch_4.engine
~/autoware_data/traffic_light_classifier/traffic_light_classifier_mobilenetv2_batch_6.engine
~/autoware_data/lidar_centerpoint/pts_backbone_neck_head_centerpoint_tiny.engine
~/autoware_data/lidar_centerpoint/pts_voxel_encoder_centerpoint_tiny.engine
~/autoware_data/traffic_light_fine_detector/tlr_car_ped_yolox_s_batch_6.engine

Installed:

sudo apt-get -y install cuda-toolkit-12-4
sudo apt-get install -y cuda-drivers-550

For the cuDNN and TensorRT, followed https://github.com/autowarefoundation/autoware/blob/c4a8feff4e59dcce34d3005b3c5e7ed89f98501c/ansible/roles/tensorrt/README.md but by using the correct amd64.env file from this version.

Also since I had the previous versions, first I had to unhold them with:

sudo apt-mark unhold \
libcudnn8 \
libnvinfer10 \
libnvinfer-plugin10 \
libnvonnxparsers10 \
libcudnn8-dev \
libnvinfer-dev \
libnvinfer-plugin-dev \
libnvonnxparsers-dev \
libnvinfer-headers-dev \
libnvinfer-headers-plugin-dev

Still I had an error:

$ sudo apt-get install -y libcudnn8=${cudnn_version} libnvinfer10=${tensorrt_version} libnvinfer-plugin10=${tensorrt_version} libnvonnxparsers10=${tensorrt_version} libcudnn8-dev=${cudnn_version} libnvinfer-dev=${tensorrt_version} libnvinfer-plugin-dev=${tensorrt_version} libnvinfer-headers-dev=${tensorrt_version} libnvinfer-headers-plugin-dev=${tensorrt_version} libnvonnxparsers-dev=${tensorrt_version}
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 libnvparsers-dev : Depends: libnvinfer-dev (= 8.6.1.6-1+cuda12.0) but 10.8.0.43-1+cuda12.8 is to be installed
E: Error, pkgProblemResolver::Resolve generated breaks, this may be caused by held packages

For this, I installed sudo apt install -y aptitude then ran:
sudo aptitude install -y libcudnn8=${cudnn_version} libnvinfer10=${tensorrt_version} libnvinfer-plugin10=${tensorrt_version} libnvonnxparsers10=${tensorrt_version} libcudnn8-dev=${cudnn_version} libnvinfer-dev=${tensorrt_version} libnvinfer-plugin-dev=${tensorrt_version} libnvinfer-headers-dev=${tensorrt_version} libnvinfer-headers-plugin-dev=${tensorrt_version} libnvonnxparsers-dev=${tensorrt_version} and it installed.

Rosbag Replay Sim Demo

✅

AWSIM Labs Demo

✅

It all works 🎊

[xmfcx]

Also tested with the sudo apt install nvidia-driver-570 as well and works without issues.

https://docs.nvidia.com/deploy/cuda-compatibility/ it seems now the newer drivers support older CUDA versions.

Backwards compatibility ensures that a newer NVIDIA driver can be used with an older CUDA Toolkit. This is implicit and most simple way of doing upgrades.

[amadeuszsz]

@xmfcx
autoware-base failed, but manual trigger done before didn't fail that early 😢

FYI, the issue dpkg: error processing package libc-bin (--configure): occurred on my local amd64 machine as well, solved via apt install qemu-system & apt install qemu-user-static but I have no idea how it applies for our runner since qemu seems configured.

I can confirm successful image build for linux/arm64 on my machine.

Potential fix for our workflow. If this is the case, please feel free to open and review this PR.

[xmfcx]

Isn't this a cache issue?

• ✅ https://github.com/autowarefoundation/autoware/actions/runs/13153842860/job/36706436068#step:5:1937
• ❌ https://github.com/autowarefoundation/autoware/actions/runs/13302868387/job/37147376203#step:5:2230

It's just doing some basic apt package installs, not even related to this PR. Or am I missing something?

@youtalk -san, maybe we should disable caching for that command?

autoware/docker/Dockerfile.base

Lines 17 to 26 in 8319f0b

	# Install apt packages and add GitHub to known hosts for private repositories
	RUN rm -f /etc/apt/apt.conf.d/docker-clean \
	&& echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' >/etc/apt/apt.conf.d/keep-cache
	RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
	apt-get update && DEBIAN_FRONTEND=noninteractive apt-get -y install --no-install-recommends \
	gosu \
	ssh \
	&& /autoware/cleanup_apt.sh \
	&& mkdir -p ~/.ssh \
	&& ssh-keyscan github.com >> ~/.ssh/known_hosts

[amadeuszsz]

@xmfcx

They upgraded latest tag recently. Initially I thought tag upgrade might resolve the issue, but now I'm up to rolling it back.