Update to runc 1.1.7 - Fixes issue with broken NVidia device support #1461

Morph-Ed · 2023-04-28T07:45:26Z

Is your feature request related to a problem? Please describe.

Photon 5 RC contains runc 1.1.4 which contains a bug regarding NVidia device registration, and as a result fails in providing NVidia CUDA support to docker containers.

See opencontainers/runc#3708

Describe the solution you'd like

I would like runc version 1.1.7 included

https://github.com/opencontainers/runc/releases/tag/v1.1.7

Describe alternatives you've considered

There might be previous versions of NVidia drivers that work. But this seems to be the most comprehensive and simple fix

Additional context

No response

ntsbtz · 2023-04-28T13:30:35Z

@Morph-Ed Working on it. Once will submit the change i will update here.

tapakund · 2023-05-02T05:25:22Z

Changes merged 772d14d

dcasota · 2023-05-05T18:10:54Z

@tapakund @ntsbtz Unfortunately something is missing. Having updated runc+containerd, there is still an issue with rootless docker.

test_user@photon-63e41f06e32c [ ~ ]$ docker run -it photon
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to apply cgroup configuration: unable to start unit "docker-b0b2a132aae95f730fec459178fc6bec45a63edf080bef8c701bbc276d029efa.scope" (properties [{Name:Description Value:"libcontainer container b0b2a132aae95f730fec459178fc6bec45a63edf080bef8c701bbc276d029efa"} {Name:Slice Value:"user.slice"} {Name:Delegate Value:true} {Name:PIDs Value:@au [1692]} {Name:MemoryAccounting Value:true} {Name:CPUAccounting Value:true} {Name:IOAccounting Value:true} {Name:TasksAccounting Value:true} {Name:DefaultDependencies Value:false}]): Permission denied: unknown.
ERRO[0000] error waiting for container:

sshedi · 2023-05-07T12:12:11Z

Do modprobe ip_tables as root and then try docker rootless.

dcasota · 2023-05-07T14:07:10Z

Hi @sshedi,

Many thanks for the weekend support !!

~~modprobe ip_tables included, the same 'create shim task' issue occurs.~~

~~Here a test script.~~
Here an install script.

# For Photon OS Docker rootless support, see
# 
# - https://vmware.github.io/photon/docs-v5/administration-guide/containers/docker-rootless-support/
# - https://github.com/vmware/photon/issues/1461
#
# Tested on Workstation17 with provisioned vm from photon-hw15-5.0-dde71ec57.x86_64.ova
# 1) login as root
# 2) save the following content e.g to a file `/tmp/photon-docker-rootless-install.sh`
# 3) run `chmod 777 /tmp/photon-docker-rootless-install.sh`
# 4) run `/tmp/photon-docker-rootless-install.sh`
# 5) If the script finished successfully, login with the user specified and rerun the script.
#

ROOTLESS_USER="test_user"
if [ `whoami | grep -o "root" | wc -l` -eq 1 ]; then
   if [ ! -f "/usr/bin/dockerd-rootless-setuptool.sh" ]; then
      # Update runc and containterd with respect to NVIDIA/nvidia-docker#1461
      tdnf update -y runc containerd
      
      tdnf install -y shadow fuse slirp4netns libslirp
      tdnf install -y docker-rootless
      useradd -m $ROOTLESS_USER
      echo Set a password for $ROOTLESS_USER.
      passwd $ROOTLESS_USER
      echo "$ROOTLESS_USER:100000:65536" >> /etc/subuid
      echo "$ROOTLESS_USER:100000:65536" >> /etc/subgid
      echo "kernel.unprivileged_userns_clone = 1" >> /etc/sysctl.d/50-rootless.conf
      chmod 644 /etc/subuid /etc/subgid /etc/sysctl.d/50-rootless.conf
      sysctl --system
      modprobe ip_tables
      echo Now login as $ROOTLESS_USER in a new for example putty window and rerun the script.
   fi
fi

# login as $ROOTLESS_USER and rerun the script
if [ `whoami | grep -o "$ROOTLESS_USER" | wc -l` -eq 1 ]; then
   if test -f "/usr/bin/dockerd-rootless-setuptool.sh"; then
      systemctl --user restart dbus
      if [ `dockerd-rootless-setuptool.sh check | grep -o "Requirements are satisfied" | wc -l` -eq 1 ]; then
         dockerd-rootless-setuptool.sh install
         if [ `cat $HOME/.bashrc | grep -o "export DOCKER_HOST=unix:///run/user/$(id -u)/docker.sock" | wc -l` -eq 0 ]; then
cat << EOF_bashrc >> $HOME/.bashrc
export PATH=/usr/bin:$PATH
export DOCKER_HOST=unix:///run/user/$(id -u)/docker.sock
EOF_bashrc
         fi
         echo The installation has finished. Check the output of hello-world.
         if [ `docker run -it hello-world | grep -o "Hello from Docker!" | wc -l` -eq 1 ]; then
           echo The installation successfully finished.
	   fi
      fi
   fi
fi

~~Here the screenoutput 1strun.~~
screen_output_1strun_root.txt
screen_output_1strun_test_user.txt

~~Here the screen output 2ndrun.~~
screen_output_2ndrun.txt

edited:
the install script works with the help of #1461 (comment) and the hint in #1474.

sshedi · 2023-05-08T10:45:55Z

Try this script and check: (run the script as root)

#!/bin/bash

set -o pipefail

source $PWD/../.lib/test-lib.sh

reg_usr="regusr"
reg_pass="changeme"

PHOTON_OS_VERSION="$(get_photon_release)"
if (( $(echo "$PHOTON_OS_VERSION < 3.0" | bc -l) )); then
  echo "*** docker-rootless is not available in Ph2 & below ***"
  exit 0
fi

fini()
{
  local ret="$?"

  tdnf erase -y docker-rootless sshpass
  modprobe -r ip_tables
  userdel -rf "${reg_usr}"

  rm test_script.sh

  if [ $ret -eq 0 ]; then
    echo -e "\n\n*** All docker-rootless tests passed ***\n\n"
  else
    echo -e "\n\n*** One or more docker-rootless tests failed ***\n\n" 1>&2
  fi

  exit "$ret"
}

trap fini EXIT

init()
{
  echo "installing docker-rootless..."
  tdnf install -y docker-rootless sshpass
  echo "installation of docker-rootless success"
  modprobe ip_tables
}

run_tests()
{
  dockerd-rootless-setuptool.sh --help

  if ! useradd -m "${reg_usr}"; then
    echo "ERROR: create user ${reg_usr} failed" 1>&2
    exit 1
  fi

  if ! echo -e "${reg_pass}\n${reg_pass}" | passwd "${reg_usr}"; then
    echo "ERROR: set password for ${reg_usr} failed" 1>&2
    exit 1
  fi

  if ! grep -qw "${reg_usr}" /etc/subuid; then
    echo "${reg_usr}:100000:65536" >> /etc/subuid
  fi

  if ! grep -qw "${reg_usr}" /etc/subgid; then
    echo "${reg_usr}:100000:65536" >> /etc/subgid
  fi

  if ! grep -qw "kernel.unprivileged_userns_clone" /etc/sysctl.d/50-rootless.conf; then
    echo "kernel.unprivileged_userns_clone = 1" >> /etc/sysctl.d/50-rootless.conf
  fi

  if ! chmod 644 /etc/subuid /etc/subgid; then
    echo "ERROR: chmod 644 /etc/subuid /etc/subgid failed" 1>&2
    exit 1
  fi

  if ! sysctl --system | grep -w "kernel.unprivileged_userns_clone"; then
    echo "ERROR: sysctl --system | grep kernel.unprivileged_userns_clone failed" 1>&2
    exit 1
  fi

  if ! dockerd-rootless-setuptool.sh --help; then
    echo "ERROR: dockerd-rootless-setuptool.sh --help failed" 1>&2
    exit 1
  fi

  # negative test, this shouldn't work if run as root
  if dockerd-rootless-setuptool.sh check; then
    echo "ERROR: dockerd-rootless-setuptool.sh check failed as root" 1>&2
    exit 1
  fi

  # create a script, run after logging in as regular user using ssh to localhost
  cat <<'EOF' > test_script.sh
#!/bin/bash

UserDbus="0"
ph_ver="$(grep 'Photon' /etc/photon-release | cut -d' ' -f4)"
if (( $(echo "$ph_ver >= 5.0" | bc -l) )); then
  UserDbus="1"
  systemctl --user restart dbus
fi

if ! dockerd-rootless-setuptool.sh check; then
  echo "ERROR: dockerd-rootless-setuptool.sh check failed" 1>&2
  exit 1
fi

if ! dockerd-rootless-setuptool.sh install; then
  echo "ERROR: dockerd-rootless-setuptool.sh install failed" 1>&2
  exit 1
fi

export PATH=/usr/bin:$PATH
export DOCKER_HOST=unix:///run/user/$(id -u)/docker.sock

img_name="test_ph_img"
img_url="photon"

if ! docker run --name ${img_name} ${img_url}; then
  echo "ERROR: docker run --name ${img_name} ${img_url} failed"
  exit 1
fi

if ! docker rm -f ${img_name}; then
  echo "ERROR: docker rm -f ${img_name} failed"
  exit 1
fi

if ! docker rmi -f ${img_url}; then
  echo "ERROR: docker rmi -f ${img_url} failed"
  exit 1
fi

if ! dockerd-rootless-setuptool.sh uninstall; then
  echo "ERROR: dockerd-rootless-setuptool.sh uninstall failed"
  exit 1
fi

if [ "$UserDbus" = "1" ]; then
  systemctl --user stop dbus dbus.socket
fi
EOF

  local SSHOPTS="-o LogLevel=ERROR -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null"
  sshpass -p "${reg_pass}" ssh -c aes256-ctr ${SSHOPTS} "${reg_usr}"@localhost 'bash -xs' < test_script.sh
}

init && run_tests

dcasota · 2023-05-08T11:31:50Z

Thanks @sshedi ******

Your pipeline test script made it clear what was missing. Pipeline scripts hopefully will become a standard. Makes life so much easier. Every enthusiast should contribute with pipeline scripts.

Wasn't aware of the dbus comment of @iwaseyusuke. The docs should be updated.

@Morph-Ed Can you retest the NVidia container?

Morph-Ed · 2023-05-08T12:32:30Z

@dcasota Happy to, is there a build process output somewhere with an ISO/OVA built with the latest code changes, so I can test? Or do I need to build it myself, or is there another way? Apologies for the naive question, first time I've been involved in Photon dev.

dcasota · 2023-05-08T13:08:08Z

Hi @Morph-Ed,

Yes sure, here a suggestion:

E.g. having started with Photon OS 5.0, update the two packages.
```
tdnf update -y runc containerd
```
In root context of the vm, e.g. with installed NVidia drivers for Cuda 12.0, run
```
docker run --rm --runtime=nvidia --gpus all nvidia/cuda:12.0.0-base-ubuntu20.04 nvidia-smi
```
For docker rootless, see the pipeline test script from @sshedi Update to runc 1.1.7 - Fixes issue with broken NVidia device support #1461 (comment).
Inside the script, there is a line with img_url="photon". Change this line to the container nvidia/cuda:12.0.0-base-ubuntu20.04. If the pipeline script successfully finishes in your environment as well, you know it's working.
The pipeline script does a cleanup. It's another purpose than for install only.
You could go with the install script suggested in Update to runc 1.1.7 - Fixes issue with broken NVidia device support #1461 (comment) as well. That install script tests if the hello-world docker container works, but doesn't uninstall anything, so you can run the NVidia container afterwards.

Both subjects might find a way into the docs.

There is no 4.0 Rev3 and 5.0 Rev1 iso so far, but make build of an iso for the 5.0 GA bits including latest updates works.

Hope this helps.

Please, confirm as soon as the issue can be closed.

I was and still am learning as well thanks to the Photon OS team + community.

Morph-Ed · 2023-05-09T10:37:42Z

Running the CUDA docker image gives me the following error

root@dde71 [ ~ ]# docker run --rm --runtime=nvidia --gpus all nvidia/cuda:12.0.0-base-ubuntu20.04 nvidia-smi
Unable to find image 'nvidia/cuda:12.0.0-base-ubuntu20.04' locally
12.0.0-base-ubuntu20.04: Pulling from nvidia/cuda
47c764472391: Pull complete
6ceabd2ff7b0: Pull complete
cd189d71cce3: Pull complete
b0b6463464d3: Pull complete
0dbb8e45df45: Pull complete
Digest: sha256:a6b1becefeac12339047245c06681edc25cf244334b9a90492925587e566a409
Status: Downloaded newer image for nvidia/cuda:12.0.0-base-ubuntu20.04
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: initialization error: driver rpc error: failed to process request: unknown.

dmesg gives:

[  186.646315] nvidia_uvm: module uses symbols nvUvmInterfaceDisableAccessCntr from proprietary module nvidia, inheriting taint.
[  186.649795] nvidia-uvm: Loaded the UVM driver, major device number 244.
[  186.650593] __vm_enough_memory: pid: 1085, comm: nvc:[driver], no enough memory for the allocation
[  186.650598] __vm_enough_memory: pid: 1085, comm: nvc:[driver], no enough memory for the allocation
[  186.650604] __vm_enough_memory: pid: 1085, comm: nvc:[driver], no enough memory for the allocation
[  186.788215] docker0: port 1(veth4e00ce3) entered disabled state
[  186.788598] device veth4e00ce3 left promiscuous mode
[  186.788603] docker0: port 1(veth4e00ce3) entered disabled state

Running the other test commands: 'photon-docker-rootless-install.sh' as root and test_user all complete ok.
Installing the NVidia drivers, the CUDA drivers, and registering the NVIDIA runtime with docker all appear to work ok:

root@dde71 [ ~ ]# nvidia-smi
Tue May  9 10:12:55 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.105.17   Driver Version: 525.105.17   CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Quadro P620         Off  | 00000000:0B:00.0 Off |                  N/A |
| 43%   48C    P0    N/A /  40W |      0MiB /  2048MiB |      1%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

My VM has 4GB RAM (all reserved) with the P620 GFX card passed through

dcasota · 2023-05-09T12:57:10Z

@Morph-Ed could you generate&attach the debug logs? see https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/user-guide.html#generating-debugging-logs

Morph-Ed · 2023-05-09T13:10:05Z

-- WARNING, the following logs are for debugging purposes only --

I0509 13:04:25.861008 1165 nvc.c:376] initializing library context (version=1.13.1, build=6f4aea0fca16aaff01bab2567adb34ec30847a0e)
I0509 13:04:25.861053 1165 nvc.c:350] using root /
I0509 13:04:25.861061 1165 nvc.c:351] using ldcache /etc/ld.so.cache
I0509 13:04:25.861067 1165 nvc.c:352] using unprivileged user 65534:65534
I0509 13:04:25.861082 1165 nvc.c:393] attempting to load dxcore to see if we are running under Windows Subsystem for Linux (WSL)
I0509 13:04:25.861141 1165 nvc.c:395] dxcore initialization failed, continuing assuming a non-WSL environment
I0509 13:04:25.865377 1171 nvc.c:278] loading kernel module nvidia
I0509 13:04:25.865473 1171 nvc.c:282] running mknod for /dev/nvidiactl
I0509 13:04:25.865503 1171 nvc.c:286] running mknod for /dev/nvidia0
I0509 13:04:25.865526 1171 nvc.c:290] running mknod for all nvcaps in /dev/nvidia-caps
I0509 13:04:25.870241 1171 nvc.c:218] running mknod for /dev/nvidia-caps/nvidia-cap1 from /proc/driver/nvidia/capabilities/mig/config
I0509 13:04:25.870336 1171 nvc.c:218] running mknod for /dev/nvidia-caps/nvidia-cap2 from /proc/driver/nvidia/capabilities/mig/monitor
I0509 13:04:25.871909 1171 nvc.c:296] loading kernel module nvidia_uvm
I0509 13:04:25.871934 1171 nvc.c:300] running mknod for /dev/nvidia-uvm
I0509 13:04:25.871993 1171 nvc.c:305] loading kernel module nvidia_modeset
I0509 13:04:25.872029 1171 nvc.c:309] running mknod for /dev/nvidia-modeset
I0509 13:04:25.872226 1172 rpc.c:71] starting driver rpc service
I0509 13:04:25.872325 1172 rpc.c:95] terminating driver rpc service
I0509 13:04:25.918221 1165 rpc.c:135] driver rpc service terminated with signal 11
I0509 13:04:25.918310 1165 nvc.c:434] shutting down library context

Searching the error text gives me this issue.

I'll see if there is anything useful to be had from that thread

Morph-Ed · 2023-05-09T14:56:38Z

Downgrading to Docker 20.10.23-1 is a potential fix, since I have 23.0.2-1.ph5 installed.

what is the best way of doing that, my repos don't have any other versions available - is there an older repo I can try to utilise?

dcasota · 2023-05-09T15:36:17Z

@sshedi does the Photon OS team recommend a NVidia-related combination of vhw + Photon OS release +Nvidia drivers/cuda/container-tool-kit ? the "testing with latest bits" approach on Photon OS 5.0 does not work because from Nvidia no support yet for docker latest.

@Morph-Ed My last setup tested was on Photon OS 4.0 rev2 with updates until August 2022 and Nvidia drivers 470.141 (Cuda 11.4). The next weeks, I might restart a research about valid combinations.

Morph-Ed · 2023-05-09T15:43:07Z

@dcasota Thanks - A colleague tells me "Photon 4 (version unknown but upgraded from 3) with NVidia 525.89.02 and CUDA 12.0 works"

I will start investigating older drivers

Morph-Ed · 2023-05-09T20:29:47Z

TLDR: A Working combo is:

photon-minimal-3.0_rev2_u2-a0f216d.iso
NVIDIA-SMI 470.141.03   Driver Version: 470.141.03   CUDA Version: 11.6
Docker version 20.10.14, build a224086
NVIDIA Container Runtime Hook version 1.10.0 commit: 7cfd3bd

Starting from the beginning... this tests a deployment of photon-minimal-3.0_rev2_u2-a0f216d.iso

All commands as root

root@photon3r2u2 [ ~ ]# tdnf update
Refreshing metadata for: 'VMware Photon Linux 3.0 (x86_64) Updates'
photon-updates                            7218    100%
Installing:
libunistring                                      x86_64                   0.9.10-2.ph3                     photon-updates              4.38M 4590714
mpfr                                              x86_64                   4.0.2-1.ph3                      photon-updates             503.20k 515280
runc                                              x86_64                   1.1.4-3.ph3                      photon-updates            10.27M 10767062
containerd                                        x86_64                   1.6.8-5.ph3                      photon-updates            87.51M 91757070
containerd-extras                                 x86_64                   1.6.8-5.ph3                      photon-updates            41.93M 43971776
libatomic_ops                                     x86_64                   7.6.8-2.ph3                      photon-updates               49.89k 51084
nettle                                            x86_64                   3.7.2-2.ph3                      photon-updates             733.64k 751248
libtasn1                                          x86_64                   4.14-2.ph3                       photon-updates             127.12k 130166
glibc-iconv                                       x86_64                   2.28-22.ph3                      photon-updates              7.13M 7473008
guile                                             x86_64                   2.0.13-3.ph3                     photon-updates            10.73M 11246897
gc                                                x86_64                   8.0.4-2.ph3                      photon-updates             517.75k 530172
autogen-libopts                                   x86_64                   5.18.16-2.ph3                    photon-updates             144.25k 147712
libnftnl                                          x86_64                   1.1.8-1.ph3                      photon-updates             219.99k 225272
libmnl                                            x86_64                   1.0.4-4.ph3                      photon-updates               27.39k 28048
libxml2-devel                                     x86_64                   2.9.11-10.ph3                    photon-updates              6.24M 6545913
zstd-libs                                         x86_64                   1.4.5-2.ph3                      photon-updates             912.98k 934896
gnutls                                            x86_64                   3.6.16-4.ph3                     photon-updates              3.58M 3758920
libmetalink                                       x86_64                   0.1.3-2.ph3                      photon-updates               68.32k 69960
libxslt                                           x86_64                   1.1.34-1.ph3                     photon-updates             386.73k 396014
attr                                              x86_64                   2.4.48-1.ph3                     photon                       87.45k 89546
libacl                                            x86_64                   2.2.53-1.ph3                     photon                       38.09k 39000
python3-typing-extensions                         noarch                   4.3.0-1.ph3                      photon-updates             157.37k 161149
docker-engine                                     x86_64                   20.10.14-8.ph3                   photon-updates            84.83M 88954866
docker-cli                                        x86_64                   20.10.14-8.ph3                   photon-updates            49.50M 51906150
gawk                                              x86_64                   4.2.1-1.ph3                      photon                      2.20M 2304491
python3-pyserial                                  noarch                   3.5-1.ph3                        photon-updates             860.77k 881426

Total installed size: 313.02M 328227840

Upgrading:
zlib                                              x86_64                   1.2.11-3.ph3                     photon-updates             124.11k 127088
xz-libs                                           x86_64                   5.2.4-2.ph3                      photon-updates             157.24k 161016
xz                                                x86_64                   5.2.4-2.ph3                      photon-updates             166.79k 170791
xmlsec1                                           x86_64                   1.2.29-6.ph3                     photon-updates           1017.23k 1041648
xerces-c                                          x86_64                   3.2.1-2.ph3                      photon-updates              4.50M 4723152
which                                             x86_64                   2.21-6.ph3                       photon-updates               33.49k 34291
vim                                               x86_64                   8.2.5169-6.ph3                   photon-updates              3.37M 3533573
util-linux-libs                                   x86_64                   2.32.1-5.ph3                     photon-updates             723.14k 740496
util-linux                                        x86_64                   2.32.1-5.ph3                     photon-updates              5.27M 5530396
tzdata                                            noarch                   2020f-2.ph3                      photon-updates              1.58M 1656520
tdnf-cli-libs                                     x86_64                   3.1.14-1.ph3                     photon-updates               70.24k 71928
tdnf                                              x86_64                   3.1.14-1.ph3                     photon-updates             313.64k 321163
systemd                                           x86_64                   239-47.ph3                       photon-updates            20.35M 21338568
sudo                                              x86_64                   1.9.5-5.ph3                      photon-updates              4.46M 4680581
sqlite-libs                                       x86_64                   3.35.5-1.ph3                     photon-updates              1.17M 1227928
shadow-tools                                      x86_64                   4.6-6.ph3                        photon-updates             162.42k 166320
shadow                                            x86_64                   4.6-6.ph3                        photon-updates              1.80M 1892281
sed                                               x86_64                   4.5-2.ph3                        photon-updates             113.48k 116199
rpm-libs                                          x86_64                   4.14.3-10.ph3                    photon-updates             942.69k 965312
rpm                                               x86_64                   4.14.3-10.ph3                    photon-updates             362.13k 370818
python3-urllib3                                   noarch                   1.25.11-3.ph3                    photon-updates             653.68k 669373
python3-six                                       noarch                   1.12.0-1.ph3                     photon-updates               59.84k 61278
python3-requests                                  noarch                   2.24.0-1.ph3                     photon-updates             316.10k 323686
python3-pyOpenSSL                                 noarch                   18.0.0-4.ph3                     photon-updates             541.96k 554965
python3-markupsafe                                x86_64                   1.1.1-1.ph3                      photon-updates               63.20k 64715
python3-deepmerge                                 noarch                   0.0.5-4.ph3                      photon-updates               25.33k 25936
python3-cryptography                              x86_64                   2.8-2.ph3                        photon-updates              1.88M 1973368
python3-cffi                                      x86_64                   1.11.5-4.ph3                     photon-updates             752.96k 771026
python3-PyYAML                                    x86_64                   5.4.1-2.ph3                      photon-updates             678.79k 695079
python3-setuptools                                noarch                   57.4.0-1.ph3                     photon-updates              5.12M 5369203
python3-xml                                       x86_64                   3.7.5-26.ph3                     photon-updates             335.36k 343410
python3-libs                                      x86_64                   3.7.5-26.ph3                     photon-updates            23.34M 24469853
python3                                           x86_64                   3.7.5-26.ph3                     photon-updates              2.95M 3088798
procps-ng                                         x86_64                   3.3.15-3.ph3                     photon-updates              1.35M 1419907
pkg-config                                        x86_64                   0.29.2-4.ph3                     photon-updates             610.52k 625172
photon-repos                                      noarch                   3.0-8.ph3                        photon-updates                 3.89k 3988
photon-release                                    noarch                   3.0-6.ph3                        photon-updates                 3.29k 3367
pcre-libs                                         x86_64                   8.44-2.ph3                       photon-updates             279.61k 286320
openssl                                           x86_64                   1.0.2zg-1.ph3                    photon-updates              4.54M 4762816
openssh-server                                    x86_64                   7.8p1-15.ph3                     photon-updates             923.25k 945408
openssh-clients                                   x86_64                   7.8p1-15.ph3                     photon-updates              3.72M 3896018
openssh                                           x86_64                   7.8p1-15.ph3                     photon-updates                    0.00b 0
open-vm-tools                                     x86_64                   12.2.0-1.ph3                     photon-updates              3.22M 3372082
nss-libs                                          x86_64                   3.44-9.ph3                       photon-updates              2.17M 2272654
nss                                               x86_64                   3.44-9.ph3                       photon-updates              2.19M 2297213
net-tools                                         x86_64                   1.60-12.ph3                      photon-updates             569.69k 583364
ncurses-terminfo                                  x86_64                   6.1-5.ph3                        photon-updates              1.94M 2034823
ncurses-libs                                      x86_64                   6.1-5.ph3                        photon-updates             625.66k 640680
ncurses                                           x86_64                   6.1-5.ph3                        photon-updates             357.89k 366475
lua                                               x86_64                   5.3.5-4.ph3                      photon-updates             648.06k 663614
linux-esx                                         x86_64                   4.19.280-1.ph3                   photon-updates            15.78M 16542265
libxml2                                           x86_64                   2.9.11-10.ph3                    photon-updates              1.52M 1593139
libtirpc                                          x86_64                   1.1.4-3.ph3                      photon-updates             213.74k 218871
libssh2                                           x86_64                   1.9.0-2.ph3                      photon-updates             238.41k 244136
libsolv                                           x86_64                   0.6.35-8.ph3                     photon-updates              1.10M 1153404
libsepol                                          x86_64                   2.8-2.ph3                        photon-updates             709.02k 726032
libselinux                                        x86_64                   2.8-2.ph3                        photon-updates             166.24k 170226
libseccomp                                        x86_64                   2.4.0-3.ph3                      photon-updates             318.09k 325720
libmspack                                         x86_64                   0.10.1alpha-2.ph3                photon-updates               79.80k 81720
libgpg-error                                      x86_64                   1.32-2.ph3                       photon-updates             209.47k 214498
libgcrypt                                         x86_64                   1.8.8-2.ph3                      photon-updates              1.20M 1253356
libstdc++                                         x86_64                   7.3.0-5.ph3                      photon-updates              1.62M 1694788
libgcc                                            x86_64                   7.3.0-5.ph3                      photon-updates               97.05k 99376
libarchive                                        x86_64                   3.3.3-9.ph3                      photon-updates             821.93k 841656
libapparmor                                       x86_64                   2.13-15.ph3                      photon-updates               85.04k 87080
less                                              x86_64                   530-2.ph3                        photon-updates             236.70k 242379
krb5                                              x86_64                   1.17-4.ph3                       photon-updates              3.32M 3479861
iputils                                           x86_64                   20180629-3.ph3                   photon-updates             278.58k 285267
iptables                                          x86_64                   1.8.3-6.ph3                      photon-updates              2.21M 2314597
iproute2                                          x86_64                   4.19.0-1.ph3                     photon-updates              2.45M 2567351
initramfs                                         x86_64                   2.0-7.ph3                        photon-updates                  50.00b 50
gzip                                              x86_64                   1.12-2.ph3                       photon-updates             146.09k 149597
grep                                              x86_64                   3.1-3.ph3                        photon-updates             241.50k 247301
glibc                                             x86_64                   2.28-22.ph3                      photon-updates              8.79M 9220831
glib                                              x86_64                   2.58.0-10.ph3                    photon-updates              3.17M 3320464
findutils                                         x86_64                   4.6.0-6.ph3                      photon-updates             518.64k 531090
file-libs                                         x86_64                   5.38-2.ph3                       photon-updates              5.70M 5974936
file                                              x86_64                   5.38-2.ph3                       photon-updates               43.18k 44214
expat-libs                                        x86_64                   2.2.9-11.ph3                     photon-updates             209.44k 214464
expat                                             x86_64                   2.2.9-11.ph3                     photon-updates               69.38k 71049
e2fsprogs-libs                                    x86_64                   1.46.5-2.ph3                     photon-updates               58.37k 59768
e2fsprogs                                         x86_64                   1.46.5-2.ph3                     photon-updates              2.12M 2220836
dracut                                            x86_64                   048-10.ph3                       photon-updates             762.72k 781026
docker                                            x86_64                   20.10.14-8.ph3                   photon-updates                    0.00b 0
dhcp-libs                                         x86_64                   4.3.5-10.ph3                     photon-updates             246.17k 252076
dhcp-client                                       x86_64                   4.3.5-10.ph3                     photon-updates              1.95M 2049093
lvm2-libs                                         x86_64                   2.02.187-2.ph3                   photon-updates              3.51M 3679984
device-mapper-libs                                x86_64                   2.02.187-2.ph3                   photon-updates             357.14k 365712
device-mapper-event-libs                          x86_64                   2.02.187-2.ph3                   photon-updates               31.05k 31800
device-mapper-event                               x86_64                   2.02.187-2.ph3                   photon-updates               45.65k 46741
lvm2                                              x86_64                   2.02.187-2.ph3                   photon-updates              2.56M 2687543
device-mapper                                     x86_64                   2.02.187-2.ph3                   photon-updates             185.81k 190267
dbus                                              x86_64                   1.13.8-3.ph3                     photon-updates             947.91k 970662
curl-libs                                         x86_64                   7.86.0-5.ph3                     photon-updates             625.18k 640184
curl                                              x86_64                   7.86.0-5.ph3                     photon-updates             272.66k 279208
cracklib-dicts                                    x86_64                   2.9.7-2.ph3                      photon-updates              8.87M 9299196
cracklib                                          x86_64                   2.9.7-2.ph3                      photon-updates               44.59k 45656
cpio                                              x86_64                   2.13-5.ph3                       photon-updates             439.97k 450532
coreutils                                         x86_64                   8.30-3.ph3                       photon-updates              5.84M 6127774
cloud-init                                        noarch                   22.4.2-4.ph3                     photon-updates              5.36M 5615669
ca-certificates-pki                               x86_64                   20190521-3.ph3                   photon-updates             799.21k 818390
ca-certificates                                   x86_64                   20190521-3.ph3                   photon-updates             749.93k 767932
bzip2-libs                                        x86_64                   1.0.8-2.ph3                      photon-updates               74.31k 76096
bzip2                                             x86_64                   1.0.8-2.ph3                      photon-updates             124.99k 127990
bash                                              x86_64                   4.4.18-4.ph3                     photon-updates              3.16M 3316212

Total installed size: 196.71M 206264725
Is this ok [y/N]:y

Downloading:
libunistring                           1385717    100%
mpfr                                    255062    100%
runc                                   3852414    100%
containerd                            29617183    100%
containerd-extras                     15055905    100%
libatomic_ops                            22749    100%
nettle                                  393176    100%
libtasn1                                 63481    100%
glibc-iconv                            2842559    100%
guile                                  3370353    100%
gc                                      231080    100%
autogen-libopts                          74808    100%
libnftnl                                 91790    100%
libmnl                                   17869    100%
libxml2-devel                           986828    100%
zstd-libs                               428781    100%
gnutls                                 1554322    100%
libmetalink                              32228    100%
libxslt                                 184276    100%
attr                                     39902    100%
libacl                                   23306    100%
python3-typing-extensions                55638    100%
docker-engine                         27991432    100%
docker-cli                            14349164    100%
gawk                                    902276    100%
python3-pyserial                        233629    100%
zlib                                     71517    100%
xz-libs                                  92342    100%
xz                                       92095    100%
xmlsec1                                 379133    100%
xerces-c                               1458141    100%
which                                    22596    100%
vim                                    1751367    100%
util-linux-libs                         348765    100%
util-linux                             2420479    100%
tzdata                                  697064    100%
tdnf-cli-libs                            36733    100%
tdnf                                    123573    100%
systemd                                6883220    100%
sudo                                   1870614    100%
sqlite-libs                             665623    100%
shadow-tools                             75692    100%
shadow                                  811814    100%
sed                                      67175    100%
rpm-libs                                367471    100%
rpm                                      88490    100%
python3-urllib3                         237519    100%
python3-six                              28202    100%
python3-requests                        119365    100%
python3-pyOpenSSL                       168921    100%
python3-markupsafe                       31236    100%
python3-deepmerge                        21713    100%
python3-cryptography                    526597    100%
python3-cffi                            261131    100%
python3-PyYAML                          222248    100%
python3-setuptools                     1569456    100%
python3-xml                              99461    100%
python3-libs                           8466374    100%
python3                                1190373    100%
procps-ng                               577735    100%
pkg-config                              268131    100%
photon-repos                             11884    100%
photon-release                           10490    100%
pcre-libs                               109153    100%
openssl                                2193728    100%
openssh-server                          461675    100%
openssh-clients                        1518322    100%
openssh                                  10936    100%
open-vm-tools                          1195358    100%
nss-libs                                929881    100%
nss                                     938711    100%
net-tools                               251011    100%
ncurses-terminfo                        726074    100%
ncurses-libs                            287194    100%
ncurses                                 211426    100%
lua                                     325096    100%
linux-esx                             11525913    100%
libxml2                                 728706    100%
libtirpc                                109811    100%
libssh2                                 111866    100%
libsolv                                 379045    100%
libsepol                                325931    100%
libselinux                               80816    100%
libseccomp                               89114    100%
libmspack                                48891    100%
libgpg-error                             98104    100%
libgcrypt                               512131    100%
libstdc++                               535125    100%
libgcc                                   56345    100%
libarchive                              391918    100%
libapparmor                              46095    100%
less                                    130134    100%
krb5                                   1488434    100%
iputils                                 121294    100%
iptables                                659984    100%
iproute2                               1275031    100%
initramfs                                11088    100%
gzip                                     83755    100%
grep                                    139024    100%
glibc                                  3985657    100%
glib                                   1357186    100%
findutils                               268205    100%
file-libs                               374111    100%
file                                     35027    100%
expat-libs                               84638    100%
expat                                    36520    100%
e2fsprogs-libs                           28645    100%
e2fsprogs                               882215    100%
dracut                                  276174    100%
docker                                    7776    100%
dhcp-libs                                74614    100%
dhcp-client                             940217    100%
lvm2-libs                              1485959    100%
device-mapper-libs                      177278    100%
device-mapper-event-libs                 21248    100%
device-mapper-event                      32369    100%
lvm2                                   1204202    100%
device-mapper                            95745    100%
dbus                                    449613    100%
curl-libs                               316929    100%
curl                                    160757    100%
cracklib-dicts                         4244904    100%
cracklib                                 26900    100%
cpio                                    170524    100%
coreutils                              2829807    100%
cloud-init                             1599824    100%
ca-certificates-pki                     360709    100%
ca-certificates                         362953    100%
bzip2-libs                               40094    100%
bzip2                                    72223    100%
bash                                   1038910    100%
Testing transaction
Running transaction
Installing/Updating: grep-3.1-3.ph3.x86_64
Installing/Updating: glibc-2.28-22.ph3.x86_64
Installing/Updating: ncurses-libs-6.1-5.ph3.x86_64
Installing/Updating: coreutils-8.30-3.ph3.x86_64
Installing/Updating: bash-4.4.18-4.ph3.x86_64
Installing/Updating: zlib-1.2.11-3.ph3.x86_64
Installing/Updating: libgcc-7.3.0-5.ph3.x86_64
Installing/Updating: openssl-1.0.2zg-1.ph3.x86_64
Installing/Updating: util-linux-libs-2.32.1-5.ph3.x86_64
Installing/Updating: bzip2-libs-1.0.8-2.ph3.x86_64
Installing/Updating: expat-libs-2.2.9-11.ph3.x86_64
Installing/Updating: xz-libs-5.2.4-2.ph3.x86_64
Installing/Updating: libstdc++-7.3.0-5.ph3.x86_64
Installing/Updating: ncurses-6.1-5.ph3.x86_64
Installing/Updating: e2fsprogs-libs-1.46.5-2.ph3.x86_64
Installing/Updating: krb5-1.17-4.ph3.x86_64
Installing/Updating: libxml2-2.9.11-10.ph3.x86_64
Installing/Updating: zstd-libs-1.4.5-2.ph3.x86_64
Installing/Updating: xz-5.2.4-2.ph3.x86_64
Installing/Updating: expat-2.2.9-11.ph3.x86_64
Installing/Updating: cracklib-2.9.7-2.ph3.x86_64
switching pw_dict to cracklib-dicts
Installing/Updating: libmnl-1.0.4-4.ph3.x86_64
Installing/Updating: libgpg-error-1.32-2.ph3.x86_64
Installing/Updating: libseccomp-2.4.0-3.ph3.x86_64
Installing/Updating: libgcrypt-1.8.8-2.ph3.x86_64
Installing/Updating: openssh-clients-7.8p1-15.ph3.x86_64
Installing/Updating: libssh2-1.9.0-2.ph3.x86_64
Installing/Updating: sqlite-libs-3.35.5-1.ph3.x86_64
Installing/Updating: python3-3.7.5-26.ph3.x86_64
'/usr/bin/python' -> '/usr/bin/python3'
Installing/Updating: python3-libs-3.7.5-26.ph3.x86_64
Installing/Updating: nss-libs-3.44-9.ph3.x86_64
Installing/Updating: python3-six-1.12.0-1.ph3.noarch
Installing/Updating: python3-xml-3.7.5-26.ph3.x86_64
Installing/Updating: pcre-libs-8.44-2.ph3.x86_64
Installing/Updating: glib-2.58.0-10.ph3.x86_64
Installing/Updating: lua-5.3.5-4.ph3.x86_64
Installing/Updating: rpm-libs-4.14.3-10.ph3.x86_64
Installing/Updating: ca-certificates-pki-20190521-3.ph3.x86_64
Installing/Updating: ca-certificates-20190521-3.ph3.x86_64
Installing/Updating: curl-libs-7.86.0-5.ph3.x86_64
Installing/Updating: python3-urllib3-1.25.11-3.ph3.noarch
Installing/Updating: python3-setuptools-57.4.0-1.ph3.noarch
Installing/Updating: nss-3.44-9.ph3.x86_64
Installing/Updating: python3-typing-extensions-4.3.0-1.ph3.noarch
Installing/Updating: python3-markupsafe-1.1.1-1.ph3.x86_64
Installing/Updating: python3-cffi-1.11.5-4.ph3.x86_64
Installing/Updating: python3-cryptography-2.8-2.ph3.x86_64
Installing/Updating: python3-pyOpenSSL-18.0.0-4.ph3.noarch
Installing/Updating: python3-requests-2.24.0-1.ph3.noarch
Installing/Updating: python3-pyserial-3.5-1.ph3.noarch
Installing/Updating: python3-PyYAML-5.4.1-2.ph3.x86_64
Installing/Updating: runc-1.1.4-3.ph3.x86_64
Installing/Updating: libnftnl-1.1.8-1.ph3.x86_64
Installing/Updating: iproute2-4.19.0-1.ph3.x86_64
Installing/Updating: shadow-tools-4.6-6.ph3.x86_64
Installing/Updating: shadow-4.6-6.ph3.x86_64
Installing/Updating: libmetalink-0.1.3-2.ph3.x86_64
Installing/Updating: libtirpc-1.1.4-3.ph3.x86_64
Installing/Updating: ncurses-terminfo-6.1-5.ph3.x86_64
Installing/Updating: libsolv-0.6.35-8.ph3.x86_64
Installing/Updating: file-libs-5.38-2.ph3.x86_64
Installing/Updating: util-linux-2.32.1-5.ph3.x86_64
Installing/Updating: docker-cli-20.10.14-8.ph3.x86_64
Installing/Updating: tdnf-cli-libs-3.1.14-1.ph3.x86_64
Installing/Updating: libapparmor-2.13-15.ph3.x86_64
Installing/Updating: findutils-4.6.0-6.ph3.x86_64
Installing/Updating: libunistring-0.9.10-2.ph3.x86_64
Installing/Updating: mpfr-4.0.2-1.ph3.x86_64
Installing/Updating: gawk-4.2.1-1.ph3.x86_64
Installing/Updating: libatomic_ops-7.6.8-2.ph3.x86_64
Installing/Updating: gc-8.0.4-2.ph3.x86_64
Installing/Updating: nettle-3.7.2-2.ph3.x86_64
Installing/Updating: libtasn1-4.14-2.ph3.x86_64
Installing/Updating: glibc-iconv-2.28-22.ph3.x86_64
Installing/Updating: guile-2.0.13-3.ph3.x86_64
Installing/Updating: autogen-libopts-5.18.16-2.ph3.x86_64
Installing/Updating: gnutls-3.6.16-4.ph3.x86_64
Installing/Updating: attr-2.4.48-1.ph3.x86_64
Installing/Updating: libacl-2.2.53-1.ph3.x86_64
Installing/Updating: systemd-239-47.ph3.x86_64
Job for systemd-udevd-kernel.socket failed.
See "systemctl status systemd-udevd-kernel.socket" and "journalctl -xe" for details.
Job for systemd-udevd-control.socket failed.
See "systemctl status systemd-udevd-control.socket" and "journalctl -xe" for details.
initramfs (re)generation triggerin 1 2 2.0-4.ph3
Installing/Updating: libsepol-2.8-2.ph3.x86_64
Installing/Updating: libselinux-2.8-2.ph3.x86_64
Installing/Updating: device-mapper-libs-2.02.187-2.ph3.x86_64
Installing/Updating: device-mapper-event-libs-2.02.187-2.ph3.x86_64
Installing/Updating: device-mapper-2.02.187-2.ph3.x86_64
Installing/Updating: device-mapper-event-2.02.187-2.ph3.x86_64
Installing/Updating: lvm2-libs-2.02.187-2.ph3.x86_64
Installing/Updating: containerd-1.6.8-5.ph3.x86_64
Installing/Updating: containerd-extras-1.6.8-5.ph3.x86_64
Installing/Updating: docker-engine-20.10.14-8.ph3.x86_64
Installing/Updating: openssh-server-7.8p1-15.ph3.x86_64
Installing/Updating: which-2.21-6.ph3.x86_64
Installing/Updating: sed-4.5-2.ph3.x86_64
Installing/Updating: pkg-config-0.29.2-4.ph3.x86_64
Installing/Updating: libxml2-devel-2.9.11-10.ph3.x86_64
Installing/Updating: libxslt-1.1.34-1.ph3.x86_64
Installing/Updating: xmlsec1-1.2.29-6.ph3.x86_64
Installing/Updating: net-tools-1.60-12.ph3.x86_64
Installing/Updating: libmspack-0.10.1alpha-2.ph3.x86_64
Installing/Updating: dhcp-libs-4.3.5-10.ph3.x86_64
Installing/Updating: dhcp-client-4.3.5-10.ph3.x86_64
Installing/Updating: cpio-2.13-5.ph3.x86_64
Installing/Updating: dracut-048-10.ph3.x86_64
Installing/Updating: initramfs-2.0-7.ph3.x86_64
initrd generation of kernel 4.19.79-1.ph3-esx will be triggered later
Installing/Updating: cloud-init-22.4.2-4.ph3.noarch
Installing/Updating: open-vm-tools-12.2.0-1.ph3.x86_64
Installing/Updating: openssh-7.8p1-15.ph3.x86_64
Installing/Updating: docker-20.10.14-8.ph3.x86_64
Installing/Updating: lvm2-2.02.187-2.ph3.x86_64
Installing/Updating: iptables-1.8.3-6.ph3.x86_64
Installing/Updating: dbus-1.13.8-3.ph3.x86_64
Installing/Updating: tdnf-3.1.14-1.ph3.x86_64
detected upgrade of tdnf, daemon-reload
Installing/Updating: file-5.38-2.ph3.x86_64
Installing/Updating: sudo-1.9.5-5.ph3.x86_64
Installing/Updating: curl-7.86.0-5.ph3.x86_64
Installing/Updating: rpm-4.14.3-10.ph3.x86_64
Installing/Updating: python3-deepmerge-0.0.5-4.ph3.noarch
Installing/Updating: iputils-20180629-3.ph3.x86_64
Installing/Updating: cracklib-dicts-2.9.7-2.ph3.x86_64
Installing/Updating: libarchive-3.3.3-9.ph3.x86_64
Installing/Updating: e2fsprogs-1.46.5-2.ph3.x86_64
Installing/Updating: procps-ng-3.3.15-3.ph3.x86_64
Installing/Updating: less-530-2.ph3.x86_64
Installing/Updating: xerces-c-3.2.1-2.ph3.x86_64
Installing/Updating: bzip2-1.0.8-2.ph3.x86_64
Installing/Updating: vim-8.2.5169-6.ph3.x86_64
Installing/Updating: photon-repos-3.0-8.ph3.noarch
Installing/Updating: photon-release-3.0-6.ph3.noarch
Installing/Updating: linux-esx-4.19.280-1.ph3.x86_64
initrd generation of kernel 4.19.280-1.ph3-esx will be triggered later
Installing/Updating: gzip-1.12-2.ph3.x86_64
Installing/Updating: tzdata-2020f-2.ph3.noarch
Removing: open-vm-tools-10.3.10-6.ph3.x86_64
Removing: lvm2-2.02.181-3.ph3.x86_64
Removing: lvm2-libs-2.02.181-3.ph3.x86_64
Removing: xmlsec1-1.2.29-2.ph3.x86_64
Removing: nss-3.44-3.ph3.x86_64
Removing: docker-18.06.2-6.ph3.x86_64
Removing: device-mapper-event-2.02.181-3.ph3.x86_64
Removing: rpm-4.14.2-5.ph3.x86_64
Removing: e2fsprogs-1.44.3-3.ph3.x86_64
Removing: tdnf-2.0.0-11.ph3.x86_64
Removing: curl-7.61.1-5.ph3.x86_64
Removing: curl-libs-7.61.1-5.ph3.x86_64
Removing: xerces-c-3.2.1-1.ph3.x86_64
Removing: sudo-1.8.23-2.ph3.x86_64
Removing: libarchive-3.3.3-3.ph3.x86_64
Removing: device-mapper-2.02.181-3.ph3.x86_64
Removing: libxml2-2.9.9-1.ph3.x86_64
Removing: libtirpc-1.1.4-1.ph3.x86_64
Removing: dbus-1.13.6-2.ph3.x86_64
Removing: iputils-20180629-2.ph3.x86_64
Removing: cloud-init-19.1-3.ph3.noarch
Removing: iptables-1.8.3-1.ph3.x86_64
Removing: libstdc++-7.3.0-4.ph3.x86_64
Removing: iproute2-4.18.0-2.ph3.x86_64
Removing: procps-ng-3.3.15-1.ph3.x86_64
Removing: device-mapper-event-libs-2.02.181-3.ph3.x86_64
Removing: device-mapper-libs-2.02.181-3.ph3.x86_64
Removing: libselinux-2.8-1.ph3.x86_64
Removing: libapparmor-2.13-7.ph3.x86_64
Removing: vim-8.1.0388-5.ph3.x86_64
Removing: python3-xml-3.7.4-3.ph3.x86_64
Removing: libssh2-1.9.0-1.ph3.x86_64
Removing: libsepol-2.8-1.ph3.x86_64
Removing: bzip2-1.0.8-1.ph3.x86_64
Removing: shadow-tools-4.6-4.ph3.x86_64
Removing: python3-PyYAML-3.13-3.ph3.x86_64
Removing: gzip-1.9-2.ph3.x86_64
Removing: less-530-1.ph3.x86_64
Removing: net-tools-1.60-11.ph3.x86_64
Removing: dhcp-client-4.3.5-6.ph3.x86_64
Removing: python3-markupsafe-1.0-3.ph3.x86_64
Removing: tdnf-cli-libs-2.0.0-11.ph3.x86_64
Removing: rpm-libs-4.14.2-5.ph3.x86_64
Removing: lua-5.3.5-2.ph3.x86_64
Removing: nss-libs-3.44-3.ph3.x86_64
Removing: libsolv-0.6.35-1.ph3.x86_64
Removing: libmspack-0.7.1alpha-1.ph3.x86_64
Removing: cracklib-dicts-2.9.6-8.ph3.x86_64
Removing: file-5.34-2.ph3.x86_64
Removing: file-libs-5.34-2.ph3.x86_64
Removing: libseccomp-2.4.0-2.ph3.x86_64
Removing: which-2.21-5.ph3.x86_64
Removing: python3-requests-2.19.1-4.ph3.noarch
Removing: python3-pyOpenSSL-18.0.0-3.ph3.noarch
Removing: ca-certificates-20190521-1.ph3.x86_64
Removing: python3-urllib3-1.23-2.ph3.noarch
Removing: python3-deepmerge-0.0.5-1.ph3.noarch
Removing: python3-setuptools-3.7.4-3.ph3.noarch
initrd of kernel 4.19.79-1.ph3-esx removed
initrd of kernel 4.19.280-1.ph3-esx removed
Removing: initramfs-2.0-4.ph3.x86_64
initramfs 2.0-4.ph3 postun
Removing: dhcp-libs-4.3.5-6.ph3.x86_64
initrd of kernel 4.19.79-1.ph3-esx removed
Removing: linux-esx-4.19.79-1.ph3.x86_64
Removing: openssh-7.8p1-5.ph3.x86_64
Removing: photon-release-3.0-3.ph3.noarch
Removing: ca-certificates-pki-20190521-1.ph3.x86_64
Removing: photon-repos-3.0-5.ph3.noarch
Removing: tzdata-2019c-1.ph3.noarch
Removing: openssh-server-7.8p1-5.ph3.x86_64
Removing: openssh-clients-7.8p1-5.ph3.x86_64
Removing: krb5-1.17-1.ph3.x86_64
Removing: dracut-048-2.ph3.x86_64
Removing: util-linux-2.32-1.ph3.x86_64
initramfs (re)generation triggerun 1 1 2.0-7.ph3
Removing: systemd-239-15.ph3.x86_64
Removing: glib-2.58.0-4.ph3.x86_64
Removing: pkg-config-0.29.2-2.ph3.x86_64
Removing: python3-cryptography-2.3.1-1.ph3.x86_64
Removing: python3-six-1.11.0-3.ph3.noarch
Removing: ncurses-terminfo-6.1-1.ph3.x86_64
Removing: python3-cffi-1.11.5-3.ph3.x86_64
Removing: python3-libs-3.7.4-3.ph3.x86_64
Removing: python3-3.7.4-3.ph3.x86_64
Removing: xz-5.2.4-1.ph3.x86_64
Removing: util-linux-libs-2.32-1.ph3.x86_64
Removing: openssl-1.0.2t-1.ph3.x86_64
Removing: sqlite-libs-3.27.2-5.ph3.x86_64
Removing: libgcrypt-1.8.5-1.ph3.x86_64
Removing: findutils-4.6.0-5.ph3.x86_64
Removing: shadow-4.6-4.ph3.x86_64
Removing: cracklib-2.9.6-8.ph3.x86_64
Removing: xz-libs-5.2.4-1.ph3.x86_64
Removing: libgpg-error-1.32-1.ph3.x86_64
Removing: e2fsprogs-libs-1.44.3-3.ph3.x86_64
Removing: ncurses-6.1-1.ph3.x86_64
Removing: expat-2.2.9-1.ph3.x86_64
Removing: cpio-2.12-4.ph3.x86_64
Removing: zlib-1.2.11-1.ph3.x86_64
Removing: sed-4.5-1.ph3.x86_64
Removing: expat-libs-2.2.9-1.ph3.x86_64
Removing: pcre-libs-8.42-1.ph3.x86_64
Removing: bzip2-libs-1.0.8-1.ph3.x86_64
Removing: libgcc-7.3.0-4.ph3.x86_64
Removing: ncurses-libs-6.1-1.ph3.x86_64
Removing: bash-4.4.18-1.ph3.x86_64
Removing: coreutils-8.30-2.ph3.x86_64
Removing: grep-3.1-2.ph3.x86_64
Removing: glibc-2.28-4.ph3.x86_64
initramfs 2.0-7.ph3 posttrans
Creating /boot/initrd.img-4.19.280-1.ph3-esx

Complete!

root@photon3r2u2 [ ~ ]# tdnf install -y build-essential wget tar

Installing:
perl                                              x86_64                   5.28.0-7.ph3                     photon-updates            56.89M 59655796
patch                                             x86_64                   2.7.6-5.ph3                      photon-updates             198.28k 203042
make                                              x86_64                   4.3-1.ph3                        photon-updates              1.36M 1422366
diffutils                                         x86_64                   3.6-3.ph3                        photon-updates              1.19M 1249815
binutils-libs                                     x86_64                   2.32-7.ph3                       photon-updates              2.64M 2764168
binutils                                          x86_64                   2.32-7.ph3                       photon-updates            24.89M 26103939
linux-api-headers                                 noarch                   4.19.280-1.ph3                   photon-updates              4.56M 4779925
autoconf                                          noarch                   2.69-8.ph3                       photon-updates              1.70M 1783277
mpc                                               x86_64                   1.1.0-2.ph3                      photon-updates             363.64k 372372
libgomp                                           x86_64                   7.3.0-5.ph3                      photon-updates             192.79k 197416
libstdc++-devel                                   x86_64                   7.3.0-5.ph3                      photon-updates            13.12M 13756707
libgomp-devel                                     x86_64                   7.3.0-5.ph3                      photon-updates             331.07k 339011
libgcc-devel                                      x86_64                   7.3.0-5.ph3                      photon-updates                132.00b 132
libgcc-atomic                                     x86_64                   7.3.0-5.ph3                      photon-updates               32.23k 33008
gcc                                               x86_64                   7.3.0-5.ph3                      photon-updates           113.53M 119045922
glibc-devel                                       x86_64                   2.28-22.ph3                      photon-updates            10.88M 11412232
bison                                             x86_64                   3.1-1.ph3                        photon                      1.75M 1836277
automake                                          noarch                   1.16.1-1.ph3                     photon                      1.37M 1436357
tar                                               x86_64                   1.30-6.ph3                       photon-updates              4.65M 4880472
wget                                              x86_64                   1.21.3-1.ph3                     photon-updates              3.04M 3185339
build-essential                                   x86_64                   0.1-3.ph3                        photon                            0.00b 0

Total installed size: 242.67M 254457573

Downloading:
perl                                  19934634   100%
patch                                   115071   100%
make                                    542823   100%
diffutils                               442661   100%
binutils-libs                           646062   100%
binutils                               8139972   100%
linux-api-headers                      1400084   100%
autoconf                                538682   100%
mpc                                     114195   100%
libgomp                                 101110   100%
libstdc++-devel                        2234212   100%
libgomp-devel                           109411   100%
libgcc-devel                              9591   100%
libgcc-atomic                            21638   100%
gcc                                   42608095   100%
glibc-devel                            3078808   100%
bison                                   584296   100%
automake                                495267   100%
tar                                    1376973   100%
wget                                   1228174   100%
build-essential                           6432   100%
Testing transaction
Running transaction
Installing/Updating: perl-5.28.0-7.ph3.x86_64
Installing/Updating: autoconf-2.69-8.ph3.noarch
Installing/Updating: automake-1.16.1-1.ph3.noarch
Installing/Updating: bison-3.1-1.ph3.x86_64
Installing/Updating: glibc-devel-2.28-22.ph3.x86_64
Installing/Updating: libgcc-atomic-7.3.0-5.ph3.x86_64
Installing/Updating: libgcc-devel-7.3.0-5.ph3.x86_64
Installing/Updating: libstdc++-devel-7.3.0-5.ph3.x86_64
Installing/Updating: libgomp-7.3.0-5.ph3.x86_64
Installing/Updating: libgomp-devel-7.3.0-5.ph3.x86_64
Installing/Updating: mpc-1.1.0-2.ph3.x86_64
Installing/Updating: gcc-7.3.0-5.ph3.x86_64
Installing/Updating: linux-api-headers-4.19.280-1.ph3.noarch
Installing/Updating: binutils-libs-2.32-7.ph3.x86_64
Installing/Updating: binutils-2.32-7.ph3.x86_64
Installing/Updating: diffutils-3.6-3.ph3.x86_64
Installing/Updating: make-4.3-1.ph3.x86_64
Installing/Updating: patch-2.7.6-5.ph3.x86_64
Installing/Updating: build-essential-0.1-3.ph3.x86_64
Installing/Updating: wget-1.21.3-1.ph3.x86_64
Installing/Updating: tar-1.30-6.ph3.x86_64

Complete!
root@photon3r2u2 [ ~ ]# tdnf install -y linux-devel
Installing:
linux                                             x86_64                   4.19.280-1.ph3                   photon-updates            27.66M 28999288
linux-devel                                       x86_64                   4.19.280-1.ph3                   photon-updates            41.79M 43825052

Total installed size:  69.45M 72824340

Downloading:
linux                                 23917408   100%
linux-devel                           12937036   100%
Testing transaction
Running transaction
Installing/Updating: linux-4.19.280-1.ph3.x86_64
initrd generation of kernel 4.19.280-1.ph3 will be triggered later
Installing/Updating: linux-devel-4.19.280-1.ph3.x86_64
(re)generate initramfs for 4.19.280-1.ph3, transfilertriggerin 2.0-7.ph3
Creating /boot/initrd.img-4.19.280-1.ph3

Complete!
root@photon3r2u2 [ ~ ]# reboot

Install NVidia Drivers

root@photon3r2u2 [ ~ ]# wget https://us.download.nvidia.com/tesla/470.141.03/NVIDIA-Linux-x86_64-470.141.03.run
root@photon3r2u2 [ ~ ]# chmod a+x ./NVIDIA-Linux-x86_64-470.141.03.run
root@photon3r2u2 [ ~ ]# ./NVIDIA-Linux-x86_64-470.141.03.run --kernel-source-path=/usr/lib/modules/`uname -r`/build --ui=none --no-questions --accept-license

SMI Test

root@photon3r2u2 [ ~ ]# nvidia-smi
Tue May  9 19:56:20 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.141.03   Driver Version: 470.141.03   CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Quadro P620         Off  | 00000000:0B:00.0 Off |                  N/A |
| 45%   48C    P0    N/A /  N/A |      0MiB /  2000MiB |      1%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Prep for NVidia Container Toolkit

root@photon3r2u2 [ ~ ]# tdnf install -y gpg
root@photon3r2u2 [ ~ ]# cd /etc/pki/rpm-gpg/
root@photon3r2u2 [ /etc/pki/rpm-gpg ]# curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | gpg --dearmor -o /etc/pki/rpm-gpg/nvidia-container-toolkit-keyring.gpg
root@photon3r2u2 [ /etc/pki/rpm-gpg ]# cat << EOF_nvct >>/etc/yum.repos.d/nvidia-container-toolkit.repo
> [libnvidia-container]
> name=libnvidia-container
> baseurl=https://nvidia.github.io/libnvidia-container/centos7/x86_64
> gpgcheck=0
> enabled=1
> EOF_nvct
root@photon3r2u2 [ /etc/pki/rpm-gpg ]# tdnf makecache

Install specific version of the NVidia container toolkit

root@photon3r2u2 [ /etc/pki/rpm-gpg ]# tdnf install nvidia-container-toolkit-1.10.0-1
Installing:
libnvidia-container1                              x86_64                   1.13.1-1                         libnvidia-container         3.12M 3269459
libnvidia-container-tools                         x86_64                   1.13.1-1                         libnvidia-container        104.39k 106899
nvidia-container-toolkit                          x86_64                   1.10.0-1                         libnvidia-container         8.75M 9170288

root@photon3r2u2 [ /etc/pki/rpm-gpg ]# systemctl restart docker
root@photon3r2u2 [ /etc/pki/rpm-gpg ]# rm /etc/yum.repos.d/nvidia-container-toolkit.repo

root@photon3r2u2 [ /etc/pki/rpm-gpg ]# docker run --rm --gpus all nvidia/cuda:11.6.2-base-ubuntu20.04 nvidia-smi
Tue May  9 20:11:47 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.141.03   Driver Version: 470.141.03   CUDA Version: 11.6     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Quadro P620         Off  | 00000000:0B:00.0 Off |                  N/A |
| 44%   49C    P0    N/A /  N/A |      0MiB /  2000MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
root@photon3r2u2 [ /etc/pki/rpm-gpg ]# docker --version
Docker version 20.10.14, build a224086
root@photon3r2u2 [ /etc/pki/rpm-gpg ]# nvidia-container-toolkit --version
NVIDIA Container Runtime Hook version 1.10.0
commit: 7cfd3bd

Morph-Ed · 2023-05-09T22:14:11Z

Cannot get Photon 5 to work (updated to use runc 1.1.7)

Tried with both

Docker version 23.0.2, build 569dd73
Docker version 20.10.14, build a224086 (taken from ph4 repo)

The NVidia driver here is from: https://us.download.nvidia.com/XFree86/Linux-x86_64/525.116.04/NVIDIA-Linux-x86_64-525.116.04.run

Which is a different driver family(?) than the one that was working in my previous experiment with Photon 3.0 (that used the Tesla driver from here https://us.download.nvidia.com/tesla/470.141.03/NVIDIA-Linux-x86_64-470.141.03.run)

photon-minimal-5.0-dde71ec57.x86_64.iso
NVIDIA-SMI 525.116.04   Driver Version: 525.116.04   CUDA Version: 12.0
Docker version xxxxxxxxxx
NVIDIA Container Runtime Hook version 1.10.0 commit: 7cfd3bd

root@photon5ga [ ~ ]# nvidia-smi
Tue May  9 22:02:04 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.116.04   Driver Version: 525.116.04   CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Quadro P620         Off  | 00000000:0B:00.0 Off |                  N/A |
| 44%   48C    P0    N/A /  40W |      0MiB /  2048MiB |      1%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
root@photon5ga [ ~ ]# docker --version
Docker version 20.10.14, build a224086
root@photon5ga [ ~ ]# docker run --rm --gpus all nvidia/cuda:11.4.0-base-ubuntu20.04 nvidia-smi
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: initialization error: driver rpc error: failed to process request: unknown.
root@photon5ga [ ~ ]# docker run --rm --gpus all nvidia/cuda:12.0.0-base-ubuntu20.04 nvidia-smi
Unable to find image 'nvidia/cuda:12.0.0-base-ubuntu20.04' locally
12.0.0-base-ubuntu20.04: Pulling from nvidia/cuda
47c764472391: Pull complete
6ceabd2ff7b0: Pull complete
cd189d71cce3: Pull complete
b0b6463464d3: Pull complete
0dbb8e45df45: Pull complete
Digest: sha256:a6b1becefeac12339047245c06681edc25cf244334b9a90492925587e566a409
Status: Downloaded newer image for nvidia/cuda:12.0.0-base-ubuntu20.04
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: initialization error: driver rpc error: failed to process request: unknown.

dmesg

[   65.063369] __vm_enough_memory: pid: 1000, comm: nvc:[driver], no enough memory for the allocation
[   65.063374] __vm_enough_memory: pid: 1000, comm: nvc:[driver], no enough memory for the allocation
[   65.063379] __vm_enough_memory: pid: 1000, comm: nvc:[driver], no enough memory for the allocation

dcasota · 2023-05-09T23:08:14Z

@Morph-Ed double check the setup result with a newer nvidia-container-toolkit, see release notes https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/release-notes.html with respect to rpc related fixes.

Morph-Ed · 2023-05-10T10:04:07Z

Failed with both a newer, and older version of docker

root@photon5ga [ ~ ]# cat /etc/photon-release
VMware Photon OS 5.0
PHOTON_BUILD_NUMBER=dde71ec57

root@photon5ga [ ~ ]# docker --version
Docker version 23.0.2, build 569dd73
root@photon5ga [ ~ ]# containerd --version
containerd github.com/containerd/containerd 1.6.17 9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
root@photon5ga [ ~ ]# runc --version
runc version 1.1.7
spec: 1.0.2-dev
go: go1.20.2
libseccomp: 2.5.4
root@photon5ga [ ~ ]# nvidia-container-toolkit --version
NVIDIA Container Runtime Hook version 1.13.1
commit: 28b70663f1a2b982e59e83bcf1844177dc745208
root@photon5ga [ ~ ]# nvidia-smi
Wed May 10 10:03:43 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.116.04   Driver Version: 525.116.04   CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Quadro P620         Off  | 00000000:0B:00.0 Off |                  N/A |
| 43%   48C    P0    N/A /  40W |      0MiB /  2048MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
root@photon5ga [ ~ ]# docker run --rm --gpus all nvidia/cuda:12.0.0-base-ubuntu20.04 nvidia-smi
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: initialization error: driver rpc error: failed to process request: unknown.

root@photon5ga [ ~ ]# docker --version
Docker version 20.10.14, build a224086
root@photon5ga [ ~ ]# containerd --version
containerd github.com/containerd/containerd 1.6.17 9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
root@photon5ga [ ~ ]# runc --version
runc version 1.1.7
spec: 1.0.2-dev
go: go1.20.2
libseccomp: 2.5.4
root@photon5ga [ ~ ]# nvidia-container-toolkit --version
NVIDIA Container Runtime Hook version 1.13.1
commit: 28b70663f1a2b982e59e83bcf1844177dc745208
root@photon5ga [ ~ ]# nvidia-smi
Wed May 10 09:56:55 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.116.04   Driver Version: 525.116.04   CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Quadro P620         Off  | 00000000:0B:00.0 Off |                  N/A |
| 43%   48C    P0    N/A /  40W |      0MiB /  2048MiB |      1%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
root@photon5ga [ ~ ]# docker run --rm --gpus all nvidia/cuda:12.0.0-base-ubuntu20.04 nvidia-smi
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: initialization error: driver rpc error: failed to process request: unknown.
root@photon5ga [ ~ ]#

Morph-Ed · 2023-05-11T12:54:48Z

Also works with Photon 3 Rev 3 Update 1

root@photon3r3u1 [ ~ ]# cat /etc/photon-release
VMware Photon OS 3.0
PHOTON_BUILD_NUMBER=05f9d3d8d
root@photon3r3u1 [ ~ ]# docker --version
Docker version 20.10.14, build a224086
root@photon3r3u1 [ ~ ]# nvidia-container-toolkit --version
NVIDIA Container Runtime Hook version 1.10.0
commit: 7cfd3bd
root@photon3r3u1 [ ~ ]# runc --version
runc version 1.1.4
spec: 1.0.2-dev
go: go1.20.4
libseccomp: 2.4.0
root@photon3r3u1 [ ~ ]# containerd --version
containerd github.com/containerd/containerd 1.6.8 9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
root@photon3r3u1 [ ~ ]# docker run --rm --gpus all nvidia/cuda:12.0.0-base-ubuntu20.04 nvidia-smi
Thu May 11 12:51:49 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.116.04   Driver Version: 525.116.04   CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Quadro P620         Off  | 00000000:0B:00.0 Off |                  N/A |
| 43%   48C    P0    N/A /  40W |      0MiB /  2048MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

I'm unable to update nvidia-container-toolkit to 1.13.1. It complains about nvidia-container-toolkit-base conflicting.

Updating from here to Photo4 using the update script, causes the CUDA docker command to fail

tigerinus · 2023-12-13T08:47:52Z

For this error:

[   65.063369] __vm_enough_memory: pid: 1000, comm: nvc:[driver], no enough memory for the allocation

Try enabling full support of memory overcommit:

sysctl -w vm.overcommit_memory=1

Morph-Ed added the enhancement label Apr 28, 2023

tapakund assigned ntsbtz Apr 28, 2023

Morph-Ed closed this as completed May 8, 2023

Morph-Ed reopened this May 8, 2023

tigerinus mentioned this issue Dec 12, 2023

Could nvidia-container-runtime work without ld.so.cache NVIDIA/nvidia-container-toolkit#71

Open

cargaona mentioned this issue May 4, 2024

driver rpc error: failed to process request: unknown. NVIDIA/nvidia-container-toolkit#236

Open

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update to runc 1.1.7 - Fixes issue with broken NVidia device support #1461

Update to runc 1.1.7 - Fixes issue with broken NVidia device support #1461

Morph-Ed commented Apr 28, 2023

ntsbtz commented Apr 28, 2023

tapakund commented May 2, 2023

dcasota commented May 5, 2023

sshedi commented May 7, 2023

dcasota commented May 7, 2023 •

edited

Loading

sshedi commented May 8, 2023 •

edited

Loading

dcasota commented May 8, 2023 •

edited

Loading

Morph-Ed commented May 8, 2023

dcasota commented May 8, 2023 •

edited

Loading

Morph-Ed commented May 9, 2023

dcasota commented May 9, 2023

Morph-Ed commented May 9, 2023

Morph-Ed commented May 9, 2023

dcasota commented May 9, 2023

Morph-Ed commented May 9, 2023

Morph-Ed commented May 9, 2023 •

edited

Loading

Morph-Ed commented May 9, 2023 •

edited

Loading

dcasota commented May 9, 2023

Morph-Ed commented May 10, 2023 •

edited

Loading

Morph-Ed commented May 11, 2023

tigerinus commented Dec 13, 2023

Update to runc 1.1.7 - Fixes issue with broken NVidia device support #1461

Update to runc 1.1.7 - Fixes issue with broken NVidia device support #1461

Comments

Morph-Ed commented Apr 28, 2023

Is your feature request related to a problem? Please describe.

Describe the solution you'd like

Describe alternatives you've considered

Additional context

ntsbtz commented Apr 28, 2023

tapakund commented May 2, 2023

dcasota commented May 5, 2023

sshedi commented May 7, 2023

dcasota commented May 7, 2023 • edited Loading

sshedi commented May 8, 2023 • edited Loading

dcasota commented May 8, 2023 • edited Loading

Morph-Ed commented May 8, 2023

dcasota commented May 8, 2023 • edited Loading

Morph-Ed commented May 9, 2023

dcasota commented May 9, 2023

Morph-Ed commented May 9, 2023

Morph-Ed commented May 9, 2023

dcasota commented May 9, 2023

Morph-Ed commented May 9, 2023

Morph-Ed commented May 9, 2023 • edited Loading

Morph-Ed commented May 9, 2023 • edited Loading

dcasota commented May 9, 2023

Morph-Ed commented May 10, 2023 • edited Loading

Morph-Ed commented May 11, 2023

tigerinus commented Dec 13, 2023

dcasota commented May 7, 2023 •

edited

Loading

sshedi commented May 8, 2023 •

edited

Loading

dcasota commented May 8, 2023 •

edited

Loading

dcasota commented May 8, 2023 •

edited

Loading

Morph-Ed commented May 9, 2023 •

edited

Loading

Morph-Ed commented May 9, 2023 •

edited

Loading

Morph-Ed commented May 10, 2023 •

edited

Loading