DNS and other errors after original network is disconnected #9603

linggao · 2021-03-03T19:46:37Z

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description
Start a container with a network, connect it to another network and disconnect it from the original network it started with. The DNS is lost for the container and it cannot even ping with an ip.

Steps to reproduce the issue:

podman network create foo-a
podman network create foo-b
podman run --name test --network foo-a -d alpine sleep 10000
podman network connect foo-b test
podman network disconnect foo-a test
podman exec -it test ping google.com
ping: bad address 'google.com'.
podman exec -it test ping 172.217.0.238
PING 172.217.0.238 (172.217.0.238): 56 data bytes
ping: sendto: Network unreachable
WARN[0000] Error resizing exec session 79bb31f01304ce475be90f0eabaf55fbb40c1fafd7bf8822419a5d3b19f3eb33: could not open ctl file for terminal resize for container a5092f2db3f45dde2e785fae5ef2ba4c70843a5e0c220744fe1f64a4e4234503: open /var/lib/containers/storage/overlay-containers/a5092f2db3f45dde2e785fae5ef2ba4c70843a5e0c220744fe1f64a4e4234503/userdata/79bb31f01304ce475be90f0eabaf55fbb40c1fafd7bf8822419a5d3b19f3eb33/ctl: no such device or address

Describe the results you received:
Please see output from step 4 and 5.

Describe the results you expected:
No errors.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:      3.1.0-dev
API Version:  3.0.0
Go Version:   go1.14.12
Git Commit:   426178a49991106ffe222f12cc42409ae78dd257-dirty
Built:        Tue Mar  2 16:08:11 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.19.6
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: Unknown
    path: /usr/local/libexec/podman/conmon
    version: 'conmon version 2.0.27-dev, commit: 7310bf13319ee8ed50799b202509bedc27b36cf8'
  cpus: 2
  distribution:
    distribution: '"rhel"'
    version: "8.3"
  eventLogger: file
  hostname: lingvs4.dev.edge-fabric.com
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 4.18.0-240.15.1.el8_3.x86_64
  linkmode: dynamic
  memFree: 2636328960
  memTotal: 8342470656
  ociRuntime:
    name: runc
    package: Unknown
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc93+dev
      commit: f245a1d1edbf545549e5a16106cf1aec356a3c7d
      spec: 1.0.2-dev
      go: go1.14.12
      libseccomp: 2.4.3
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_MKNOD,CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    selinuxEnabled: true
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 2146758656
  swapTotal: 2146758656
  uptime: 316h 24m 18.01s (Approximately 13.17 days)
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 3
    paused: 0
    running: 3
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageStore:
    number: 10
  runRoot: /var/run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.0.0
  Built: 1614722891
  BuiltTime: Tue Mar  2 16:08:11 2021
  GitCommit: 426178a49991106ffe222f12cc42409ae78dd257-dirty
  GoVersion: go1.14.12
  OsArch: linux/amd64
  Version: 3.1.0-dev

Package info (e.g. output of rpm -q podman or apt list podman):

podman is built from the latest master.

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes/No

Additional environment details (AWS, VirtualBox, physical, etc.):

The text was updated successfully, but these errors were encountered:

baude · 2021-03-04T13:31:29Z

what is 172.217.0.238 ?

baude · 2021-03-04T13:40:10Z

Ok, here is the diagnosis. There are two bugs here.

When I reproduce this, I think the dns is somewhat expected? but maybe actually a bug. When connecting to the first network, you get a /etc/resolv.conf pointing to the gateway of the first network. when you connect to the second network, that is unchanged. when we remove the first network, /etc/resolv.conf is unchanged.

There seems to be a secondary problem where when adding a second network, no route is defined. So when the first network is removed, the container seems to loose it's default route.

baude · 2021-03-04T13:49:43Z

additionally, if you disconnect the first network and then connect the second, a default route is defined. (and again, the resolve is unchanged).

@mheon lets talk about this one ...

linggao · 2021-03-04T16:47:52Z

@baude 172.217.0.238 is the IP address returned when I ping google.com.

jarcher · 2021-03-05T16:43:33Z

What are all of the flags you are using for managing the containers with the hzn client? are yall using the network-alias, link and/or dns flags from the docker version?

linggao · 2021-03-06T00:10:51Z

@jarcher we are using go-dockerclient to talk to the API endpoint. We definitely use network alias. Not dns or link. Here is an example of our container using docker inspect.

    {
        "Id": "f7c60861c01a40106b3877a3634bd8783d9ddad02b5455b3ec075ea9f75d07ad",
        "Created": "2021-03-02T17:38:11.066186717Z",
        "Path": "ping",
        "Args": [
            "google.com"
        ],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 2264433,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2021-03-02T17:38:12.074663553Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:e651fe6a248e9f96352e83878fdf943969e3eea2032b3b135458a1988161ff2c",
        "ResolvConfPath": "/var/lib/docker/containers/f7c60861c01a40106b3877a3634bd8783d9ddad02b5455b3ec075ea9f75d07ad/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/f7c60861c01a40106b3877a3634bd8783d9ddad02b5455b3ec075ea9f75d07ad/hostname",
        "HostsPath": "/var/lib/docker/containers/f7c60861c01a40106b3877a3634bd8783d9ddad02b5455b3ec075ea9f75d07ad/hosts",
        "LogPath": "",
        "Name": "/anaxsquad_lingPodmanDep1_0.0.1_cec0af8e-1fff-4cfd-a762-8bd2befd689d-ling_podman_dep1",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/tmp/testdata:/tmp/mydata:ro",
                "myvolume1:/tmp/mydata2",
                "/tmp/hzndev:/tmp/hzndev",
                "/tmp/hzndev/auth/anaxsquad_lingPodmanDep1_0.0.1_cec0af8e-1fff-4cfd-a762-8bd2befd689d:/ess-auth:ro",
                "/tmp/hzndev/auth/SSL/cert:/ess-cert:ro"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "syslog",
                "Config": {
                    "tag": "workload-anaxsquad_lingpodmandep1_0.0.1_cec0af8e-1fff-4cfd-a762-8bd2befd689d_ling_podman_dep1"
                }
            },
            "NetworkMode": "anaxsquad_lingPodmanDep1_0.0.1_cec0af8e-1fff-4cfd-a762-8bd2befd689d",
            "PortBindings": {
                "7777/udp": [
                    {
                        "HostIp": "127.0.0.1",
                        "HostPort": ""
                    }
                ],
                "8888/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": ""
                    }
                ]
            },
            "RestartPolicy": {
                "Name": "always",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": [
                "SYS_ADMIN",
                "SYS_TIME"
            ],
            "CapDrop": null,
            "Capabilities": null,
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": null,
            "GroupAdd": [
                "1007"
            ],
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "Tmpfs": {
                "/app": ""
            },
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 4294967296,
            "NanoCpus": 1500000000,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [
                {
                    "PathOnHost": "/dev/bus/usb/001/001",
                    "PathInContainer": "/dev/bus/usb/001/003",
                    "CgroupPermissions": "rwm"
                }
            ],
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "KernelMemory": 0,
            "KernelMemoryTCP": 0,
            "MemoryReservation": 0,
            "MemorySwap": 8589934592,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/304d0cabbae1c8c84bcb0e98bc2d94e659d719c2a897d16486f0fc30fa41134f-init/diff:/var/lib/docker/overlay2/38a1167ffa9609029aaf061988641c803276d536c479c18879ca5533e43954a8/diff:/var/lib/docker/overlay2/394a8b350f89a16ebf4fee478941335cd28337d20d51b7723fd9c24367eabd71/diff",
                "MergedDir": "/var/lib/docker/overlay2/304d0cabbae1c8c84bcb0e98bc2d94e659d719c2a897d16486f0fc30fa41134f/merged",
                "UpperDir": "/var/lib/docker/overlay2/304d0cabbae1c8c84bcb0e98bc2d94e659d719c2a897d16486f0fc30fa41134f/diff",
                "WorkDir": "/var/lib/docker/overlay2/304d0cabbae1c8c84bcb0e98bc2d94e659d719c2a897d16486f0fc30fa41134f/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/tmp/hzndev",
                "Destination": "/tmp/hzndev",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/tmp/hzndev/auth/anaxsquad_lingPodmanDep1_0.0.1_cec0af8e-1fff-4cfd-a762-8bd2befd689d",
                "Destination": "/ess-auth",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/tmp/hzndev/auth/SSL/cert",
                "Destination": "/ess-cert",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/tmp/testdata",
                "Destination": "/tmp/mydata",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "volume",
                "Name": "myvolume1",
                "Source": "/var/lib/docker/volumes/myvolume1/_data",
                "Destination": "/tmp/mydata2",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": ""
            }
        ],
        "Config": {
            "Hostname": "f7c60861c01a",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "7777/udp": {},
                "8888/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "HZN_ESS_API_ADDRESS=/tmp/hzndev/essapi.sock",
                "HZN_ESS_AUTH=/ess-auth/auth.json",
                "HZN_ESS_CERT=/ess-cert/cert.pem",
                "HZN_PATTERN=",
                "HZN_ARCH=amd64",
                "HZN_PRIVILEGED=false",
                "HW_WHO=World Dep1",
                "HZN_DEVICE_ID=lingvs2",
                "HZN_ESS_API_PROTOCOL=secure-unix",
                "HZN_ESS_API_PORT=0",
                "HZN_CPUS=4",
                "HZN_HOST_IPS=127.0.0.1,10.167.11.74,169.55.186.244,172.17.0.1",
                "HZN_EXCHANGE_URL=https://cp-console.ieam-roks-stg-70ea81cdef68a2eb78ece6d890b7dad3-0000.us-south.containers.appdomain.cloud/edge-exchange/v1",
                "HZN_RAM=32156",
                "HZN_ORGANIZATION=anaxsquad",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": [
                "google.com"
            ],
            "Image": "linggao/ling_podman_dep1_amd64:0.0.1",
            "Volumes": {
                "/ess-auth": {},
                "/ess-cert": {},
                "/tmp/hzndev": {},
                "/tmp/mydata": {},
                "/tmp/mydata2": {}
            },
            "WorkingDir": "/",
            "Entrypoint": [
                "ping"
            ],
            "OnBuild": null,
            "Labels": {
                "openhorizon.anax.agreement_id": "anaxsquad_lingPodmanDep1_0.0.1_cec0af8e-1fff-4cfd-a762-8bd2befd689d",
                "openhorizon.anax.deployment_description_hash": "d-2JFGJPZWhXCQhFqwyE-kam1zU=",
                "openhorizon.anax.dev_service": "true",
                "openhorizon.anax.infrastructure": "",
                "openhorizon.anax.service_name": "ling_podman_dep1",
                "openhorizon.anax.variation": ""
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "aabb3c5a05efe168c36dc93c5cd102db6f0fb5d335835f27ebcaff87db7c842a",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "7777/udp": [
                    {
                        "HostIp": "127.0.0.1",
                        "HostPort": "32917"
                    }
                ],
                "8888/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "32931"
                    }
                ]
            },
            "SandboxKey": "/var/run/docker/netns/aabb3c5a05ef",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "anaxsquad_lingPodmanDep1_0.0.1_cec0af8e-1fff-4cfd-a762-8bd2befd689d_2827da80bb7a652e540135cf416823ebfc5c94df16ee2121896a097be4d94f93": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "ling_podman_dep1",
                        "f7c60861c01a"
                    ],
                    "NetworkID": "8b1c5c67719541c282c4fa5e73882040e229dcf3dc7047db2e84bc712443fd5f",
                    "EndpointID": "e0413f7bb9b797c7b48b7966f03be664c5ac507007247b00e9634e3f71e8f272",
                    "Gateway": "172.28.0.1",
                    "IPAddress": "172.28.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:1c:00:02",
                    "DriverOpts": null
                },
                "anaxsquad_lingPodmanDep1_0.0.1_cec0af8e-1fff-4cfd-a762-8bd2befd689d_anaxsquad_lingPodmanDep2_0.0.1_b3bad5e9-48a0-436d-90f1-4fe8fe708cb5": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "ling_podman_dep1",
                        "f7c60861c01a"
                    ],
                    "NetworkID": "83b8688c7293361058d2bdcf7e57e5228776eefd7ecdaeb0e146b2831a5d3d39",
                    "EndpointID": "d8b8ffc8ef52051b37eb4283119cef95fd327c199009da2c2f56d5ea8681f3e4",
                    "Gateway": "172.29.0.1",
                    "IPAddress": "172.29.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:1d:00:02",
                    "DriverOpts": null
                }
            }
        }
    }
]

mccv1r0 · 2021-03-10T20:45:26Z

Issue opened in cni plugins; containernetworking/plugins#601

github-actions · 2021-04-10T00:07:41Z

A friendly reminder that this issue had no activity for 30 days.

umohnani8 · 2021-09-30T16:57:53Z

the issue in cni plugins was fixed and closed, is this still an issue here @baude @mheon @linggao?

mheon · 2021-09-30T17:00:36Z

Unsure. It is definitely a target of our current network rewrite efforts, though.

linggao · 2021-09-30T17:23:29Z

@umohnani8 I am okay to close it if the scenario described in the issue is working. Namely:

podman network create foo-a
podman network create foo-b

podman run --name test --network foo-a -d alpine sleep 10000

podman network connect foo-b test
podman network disconnect foo-a test

podman exec -it test ping google.com

I do not have the env to test it now.

github-actions · 2021-11-13T00:04:01Z

A friendly reminder that this issue had no activity for 30 days.

baude · 2022-01-17T18:51:06Z

this should be fixed with aardvark and podman 4.0 release

baude · 2022-02-04T18:39:03Z

works with podman 4

baude · 2022-02-04T18:40:04Z

i want to double check this one

The `podman network connect` and `podman network disconnect` commands give containers access to different networks than the ones they were created with; these networks can also have DNS servers associated with them. Until now, however, we did not modify resolv.conf as network membership changed. With this PR, `podman network connect` will add any new nameservers supported by the new network to the container's /etc/resolv.conf, and `podman network disconnect` command will do the opposite, removing the network's nameservers from `/etc/resolv.conf`. Fixes containers#9603 Signed-off-by: Matthew Heon <matthew.heon@pm.me>

openshift-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Mar 3, 2021

linggao changed the title ~~DNS error after original network is disconnected~~ DNS and other errors after original network is disconnected Mar 3, 2021

github-actions bot added the stale-issue label Apr 10, 2021

Luap99 added the network Networking related issue or feature label Jun 21, 2021

umohnani8 removed the stale-issue label Sep 30, 2021

github-actions bot added the stale-issue label Nov 13, 2021

baude added the 4.0 label Jan 17, 2022

github-actions bot removed the stale-issue label Jan 18, 2022

mheon self-assigned this Feb 3, 2022

baude closed this as completed Feb 4, 2022

baude reopened this Feb 4, 2022

baude assigned Luap99 and unassigned mheon Feb 4, 2022

mheon mentioned this issue Feb 9, 2022

Modify /etc/resolv.conf when connecting/disconnecting #13191

Merged

openshift-merge-robot closed this as completed in #13191 Feb 10, 2022

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 21, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DNS and other errors after original network is disconnected #9603

DNS and other errors after original network is disconnected #9603

linggao commented Mar 3, 2021 •

edited

Loading

baude commented Mar 4, 2021

baude commented Mar 4, 2021

baude commented Mar 4, 2021

linggao commented Mar 4, 2021

jarcher commented Mar 5, 2021

linggao commented Mar 6, 2021 •

edited

Loading

mccv1r0 commented Mar 10, 2021

github-actions bot commented Apr 10, 2021

umohnani8 commented Sep 30, 2021

mheon commented Sep 30, 2021

linggao commented Sep 30, 2021 •

edited

Loading

github-actions bot commented Nov 13, 2021

baude commented Jan 17, 2022

baude commented Feb 4, 2022

baude commented Feb 4, 2022

DNS and other errors after original network is disconnected #9603

DNS and other errors after original network is disconnected #9603

Comments

linggao commented Mar 3, 2021 • edited Loading

baude commented Mar 4, 2021

baude commented Mar 4, 2021

baude commented Mar 4, 2021

linggao commented Mar 4, 2021

jarcher commented Mar 5, 2021

linggao commented Mar 6, 2021 • edited Loading

mccv1r0 commented Mar 10, 2021

github-actions bot commented Apr 10, 2021

umohnani8 commented Sep 30, 2021

mheon commented Sep 30, 2021

linggao commented Sep 30, 2021 • edited Loading

github-actions bot commented Nov 13, 2021

baude commented Jan 17, 2022

baude commented Feb 4, 2022

baude commented Feb 4, 2022

linggao commented Mar 3, 2021 •

edited

Loading

linggao commented Mar 6, 2021 •

edited

Loading

linggao commented Sep 30, 2021 •

edited

Loading