podman 3.0.1 rootless network issues: Connection reset by peer

[ms42Q]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

I want run a django application in a rootless podman environment.
The application starts up and even connects to a DB that is running in the same named network.
It even looks like the port is exposed. But when i do a curl localhost:8080 i get a (56) Recv failure: Connection reset by peer

The setup worked with podman 2.2.1 it only fails with 3.0.1

Steps to reproduce the issue:

1. create named network

2. run application with -p "8080:8080"

3. curl localhost:800

when i execute podman exec -it app curl localhost:8080 it shows what i expext.

This is the systemd unit I am using:
I removed all the application specific values.

[Unit]
Description=Podman app.service
Wants=network.target
Requires=app-db.service
After=app-db.service
StartLimitIntervalSec=10
StartLimitBurst=5


[Service]
User=podman-user
Group=podman-user
Environment=PODMAN_SYSTEMD_UNIT=%n
TimeoutStartSec=3m

Restart=on-failure
ExecStartPre=/usr/bin/podman run \
        --network=app-network \
    registry.gitlab.com/path/to/image:main \
    python manage.py migrate
ExecStart=/usr/bin/podman run \
    --name app \
        --network=app-network \
            -p "8080:8080" \
    registry.gitlab.com/path/to/image:main \
    waitress-serve --port=8080 app.wsgi:application
ExecStop=/usr/bin/podman stop \
    --ignore \
    -t 10 \
    app
ExecStopPost=/usr/bin/podman rm \
    --ignore \
    -f \
    app

[Install]
WantedBy=multi-user.target default.target

Describe the results you received:

When everything is up and i do a curl localhost:8080 i get a connection reset.

The output of netstat is a bit weird:

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      484/sshd            
tcp6       0      0 :::8080                 :::*                    LISTEN      1324/containers-roo 
tcp6       0      0 :::22                   :::*                    LISTEN      484/sshd            

In the podman 2.2.1 setup there is an entry for 0.0.0.0:8080

There is also something else going on. When I manually restart the container with podman restart i first get a 0.0.0.0:8080 is already in use message and then a lot of these:

ERRO[0000] error joining network namespace for container 4f2e652d70422c0d01da082b41ec80e40bc4419b68cb8bd615d80852d61d92e2: error retrieving network namespace at /proc/3210/ns/net: failed to Statfs "/proc/3210/ns/net": no such file or directory

And then I have trouble bringing the container up agasin

Describe the results you expected:

I expect that the exposed port gives me access to the application

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:      3.0.1
API Version:  3.0.0
Go Version:   go1.14
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/amd64

Output of podman info --debug:

(paste your output here)
host:
  arch: amd64
  buildahVersion: 1.19.4
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.26, commit: '
  cpus: 2
  distribution:
    distribution: debian
    version: "10"
  eventLogger: journald
  hostname: debian
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 4.19.0-14-amd64
  linkmode: dynamic
  memFree: 878088192
  memTotal: 2091941888
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 0.18.1-7931a-dirty
      commit: 7931a1eab0590eff4041c1f74e2844b297c31cea
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: true
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    selinuxEnabled: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 2144333824
  swapTotal: 2144333824
  uptime: 28m 23.71s
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageStore:
    number: 0
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.0.0
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.14
  OsArch: linux/amd64
  Version: 3.0.1

Package info (e.g. output of rpm -q podman or apt list podman):

podman/unknown,now 100:3.0.1-2 amd64 [residual-config]
podman/unknown 100:3.0.1-2 arm64
podman/unknown 100:3.0.1-2 armhf
podman/unknown 100:3.0.1-2 ppc64el
podman/unknown 100:3.0.1-2 s390x

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

I also tried all of the suggestions I could find that are listed in issues that seemed to be related.
Like adding --net=slirp4netns:port_handler=slirp4netns

Additional environment details (AWS, VirtualBox, physical, etc.):

qemu+KVM

[millerthegorilla]

I have this problem on fedora silverblue running a pod. There is no network connectivity to the pod on update to:

podman 2:2.2.1-1.fc33 -> 2:3.0.1-1.fc33
podman-plugins 2:2.2.1-1.fc33 -> 2:3.0.1-1.fc33

so I performed an rpm-ostree rollback and its working again. I have been developing using this pod, and want to start using a production server, with the pod, and installing coreos is going to bring the latest podman which will stop my pod from functioning, unless this is addressed, and I won't be able to rollback on that new install.

I see from the update preview the following referenced:

FEDORA-2021-fb466fb623

which discusses a moderate severity bug at the following:

https://bugzilla.redhat.com/show_bug.cgi?id=1919391

which describes:

When specifying the plugin to load in the `type` field in the network configuration, it is possible to use special elements such as "../" separators to reference binaries elsewhere on the system.

So as a first guess, I am going to try recreating the containers, to see if they are then compatible with the pod and networking.

Edit... Recreated pod and containers with new version of podman installed, and still connection reset.

Also... I notice the issue thread is about a podman setup using a specific network, where I am simply using a created pod with a standard published port and containers created with --pod switch. Should I open a separate issue?

[ms42Q]

@millerthegorilla Thanks so much for confirming the issue!

It is a pity that 2.2.1 was removed from the repositories. My environment does not allow me to build locally which makes the downgrade a hassle. Do you know if there is any other workaround for his issue?

[millerthegorilla]

I can't believe it has been removed from repositories, without the new version being tested properly.

I mean it's really screwed me over as podman doesn't work at all with the new version.

I haven't found a fix for using a pod. My next attempt would be to try running containers with a podman network, but if I understand the original post in this issue that isn't working either.

I hope that there is some involvement from a member of the team on this issue soon.

I have used the command:

ostree admin pin 0

to pin the deployment with the working podman version, so I hope to not lose the functionality. It exposes a real flaw with ostree systems, in that I have a load of necessary updates, but can't apply them without updating podman, and there is no way that I can find to pin the specific package to the working version.

[Luap99]

Do you have slirp4netns installed? The output from podman info has no information about it.

[Luap99]

I found the problem. Commit 479fc22 is missing in v3.0

[millerthegorilla]

genius. nice one. Hopefully its not too difficult to remedy?

[Luap99]

@mheon If you do the backports for 3.0.2 please add commit 479fc22.

[mheon]

Ack. Expecting a release around Thursday. I can do the backport Monday to prepare.

…

On Sat, Feb 27, 2021 at 11:04 Paul Holzinger ***@***.***> wrote: @mheon <https://github.com/mheon> If you do the backports for 3.0.2 please add commit 479fc22 <479fc22> . — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#9532 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB3AOCEYTW4OO65HPXN5OZTTBEJZLANCNFSM4YI6TY5Q> .

[mheon]

We should also get a test written for this, so we don't regress in the future.

[Luap99]

This commit includes a test

[mheon]

Ah, excellent.

[millerthegorilla]

@mheon is there likely to be an update soon? I am relying on podman and am using fedora silverblue, and so I can't update until the new podman version is released. So, I now have a whole host of security updates waiting, amongst others.

[mheon]

Yesterday we looked at how much had landed and decided to cut a 3.1.0 instead of doing extensive backporting. I'm going to cut an RC on Monday, final release Wednesday-Thursday of next week. Sorry for the delay here.

[millerthegorilla]

Ok, thanks. Do you know if there is a way to get podman working in the meantime?

[millerthegorilla]

May I ask, is podman a sensible solution for use in production environments? I have been intending to use podman in production, but if its going to have week plus down time then its not sensible for production. I will have to think about using docker instead, which I will have to learn to use. I wanted to use podman because it is rootless.

[mheon]

We're commercially supporting it in RHEL, so I would certainly say it is. We had something slip through our test suite here, but that's not exactly typical.

[millerthegorilla]

oh. ok. Thanks. Is there a way to monkey patch the 3.0.1 podman until the new version is released?
Actually, because I'm on silverblue, I don't suppose there is, as the system file tree is immutable. damn.

[millerthegorilla]

@mheon many thanks for your help, I have been instructed on how to pin a package version, so have been able to update.

[cgwalters]

Actually, because I'm on silverblue, I don't suppose there is, as the system file tree is immutable. damn.

No 😄 The goal is to empower you and others, not restrict you. Local overrides are for exactly this type of case.

[millerthegorilla]

Cool. I've learnt something really useful through this issue, that will help when I'm running a coreos server. I've found the instructions and tutorials for rpm-ostree a bit thin on the ground, or I'm searching badly, so the ability to pin the package and manage this use case is really valuable. Look forward to the podman release... thanks for all your work!

[millerthegorilla]

Hey @mheon I have tried upgrading using the rpm-ostree reset method, to podman version 3.1.0-rc1 and I am still getting connection reset by peer. Is that right? Or should I try a later version?
https://koji.fedoraproject.org/koji/buildinfo?buildID=1720774

[sayap]

Version 3.1.0-rc1 fixes the connection reset issue for me. This is on a gentoo host.

[millerthegorilla]

It doesn't fix it for me. I think I should probably open a new bug report.

[millerthegorilla]

I have just tried resetting my base and installing podman version 3.1.0-rc1 again, and it still gives me:

curl: (56) Recv failure: Connection reset by peer

[ms42Q]

Is anyone still working on this? I already had to delay a production release for almost 3 weeks because of this. I don't want to nag but if this is taking another week I need to revert everything to docker i guess.

Alternatively, can anyone tell me how I can revert to the 2.0.4 release on debian? or can someone maybe push podman 2.0. back into https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/Debian_10

I just looked into #9746

When will a new release with a fix be pushed to the debian repo?

[mheon]

You could try using Podman from the official Debian repos - I believe it's only in testing now, but they should have both 2.2.x and 3.0.x .deb files available. We're starting to move away from the Kubic repos because of things like this.

[mheon]

As for a fresh release, we're expecting one more RC on Monday, and then a final release later next week.

[ms42Q]

thanks @mheon !

[ms42Q]

but they should have both 2.2.x and 3.0.x .deb files available.

unfortunately debian testing also only has 3.0.1 available.

[Luap99]

Podman v3.1.0 is released, closing.