podman: publishing huge amount of port requires modification of ulimit

[jfilak]

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

Description
It is not possible to publish 1000 ports via -p 1000-2000:1000-2000 without changing my ulimit -n which is by default 1024 (2048 was not enough in my bash, 4096 works for me). I am not sure if it is a bug, but it is a deviation from docker behaviour. Perhaps, I could write a blog post but this issue might server the documentation purpose as well.

Steps to reproduce the issue:

1. podman run -it -p 1000-2000:1000-2000 --rm --name manyports centos /bin/bash

Describe the results you received:

I get the following error:

ERRO[0000] `iptables -t filter -D FORWARD -s 10.88.0.2 ! -o 10.88.0.2 -j ACCEPT` failed:   (pipe2: too many open files) 
ERRO[0000] Error deleting network: pipe2: too many open files 
ERRO[0000] Error while removing pod from CNI network "crio-bridge": pipe2: too many open files 
ERRO[0000] unable to cleanup network for container 2a082c5f7e06eb1242d6627b915617048d70eb43d8fc9710fabdad4f713cb96a: "error tearing down CNI namespace configuration for container 2a082c5f7e06eb1242d6627b915617048d70eb43d8fc9710fabdad4f713cb96a: pipe2: too many open files" 
cannot listen on the TCP port: listen tcp4 :1632: socket: too many open files

Describe the results you expected:

The container is started.

Additional information you deem important (e.g. issue happens only occasionally):

The user must change ulimit -n to be able to publish many ports.

Output of podman version:

Version:       0.8.3
Go Version:    go1.10.3
OS/Arch:       linux/amd64

Output of podman info:

host:
  Conmon:
    package: podman-0.8.3-1.git9d09a4d.fc28.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 1.12.0-dev, commit: c72f69ce3f3fd7e46dced78dd99b67ab5441441e-dirty'
  MemFree: 12209209344
  MemTotal: 16547180544
  OCIRuntime:
    package: runc-1.0.0-50.dev.git20aff4f.fc28.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.0'
  SwapFree: 8346660864
  SwapTotal: 8346660864
  arch: amd64
  cpus: 4
  hostname: brql33992173a
  kernel: 4.17.18-200.fc28.x86_64
  os: linux
  uptime: 17m 19.44s
insecure registries:
  registries: []
registries:
  registries:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ContainerStore:
    number: 0
  GraphDriverName: overlay
  GraphOptions:
  - overlay.mountopt=nodev
  - overlay.override_kernel_check=true
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
  ImageStore:
    number: 1
  RunRoot: /var/run/containers/storage

Additional environment details (AWS, VirtualBox, physical, etc.):

physical Fedora 28

[giuseppe]

I think this is caused by the new code we added to keep a port busy and not let other applications re-use it.

[mheon]

For non-rootless code, we can set some high default ulimits in Podman and Conmon when we start, which should avoid this. I believe Docker defaults to somewhere around a hundred thousand?

…

On Tue, Aug 28, 2018, 04:50 Giuseppe Scrivano ***@***.***> wrote: I think this is caused by the new code we added to keep a port busy and not let other applications re-use it. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1357 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHYHCPgEF-y7IBoDzhP5g8u7Irzbsu_Kks5uVQRUgaJpZM4WO5uv> .

[giuseppe]

should be fine to set up to 1048576. Although that is a lot of memory even without hitting that hard limit, just to keep the fd's open for some valid use cases like forwarding a lot of ports. Should we instead have a maximum and after that don't try to keep the port busy?

[giuseppe]

(any higher value must be tweaked first through sysctl fs.nr_open

[mheon]

If we think memory usage could be a problem, it might be a good idea to have a config flag to disable opening FDs - seems easier to explain to users

[baude]

we could calculate the number of ports to be open easy enough no?

[mheon]

Sure, but I like an on/off switch versus turning off expected behavior after a certain threshold

[baude]

well what i was implying is that we could set the ulimit to something greater than the number of ports ...

[mheon]

This should be fixed via #1437