Container not accessible from the network with Podman instead of Docker

[itodorovic]

Issue Description

When running the container with podman instead of docker, container is not visible from other devices on the network.
This seems to be discussed in upstream project:
b-data/data-science-devcontainers#1
b-data/data-science-devcontainers#1 (comment)
I'm using CDI.

To Reproduce
Steps to reproduce the behavior:
Run with:
podman run --device nvidia.com/gpu=all -d -it -p 8848:8888 -v $(pwd)/data:/home/jovyan/work -e GRANT_SUDO=yes -e JUPYTER_ENABLE_LAB=yes -e JUPYTER_TOKEN=${JUPYTER_TOKEN} --user root --restart always --name gpu-jupyter cschranz/gpu-jupyter:v1.6_cuda-12.0_ubuntu-22.04

Environment

Operating System:
RHEL 8

NVIDIA GPU and CUDA version Details:

# nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Wed_Nov_22_10:17:15_PST_2023
Cuda compilation tools, release 12.3, V12.3.107
Build cuda_12.3.r12.3/compiler.33567101_0

# nvidia-smi
Mon Apr  1 12:25:55 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.14              Driver Version: 550.54.14      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Tesla P6                       Off |   00000000:13:00.0 Off |                    0 |
| N/A   34C    P0             26W /   90W |    8128MiB /  15360MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A     77564      C   /server                                      8126MiB |
+-----------------------------------------------------------------------------------------+

[benz0li]

[...] container is not visible from other devices on the network

Can you curl -v http://127.0.0.1:8848 from the same device (i.e. the container host)?

If so, then port 8848 is blocked (firewall?) and this issue is not related to the container.