Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failing e2e tests #8309

Closed
aodhan-domhnaill opened this issue Nov 12, 2020 · 14 comments
Closed

Failing e2e tests #8309

aodhan-domhnaill opened this issue Nov 12, 2020 · 14 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@aodhan-domhnaill
Copy link

/kind bug

Description

Make test is failing integ tests.

/home/aidan/foss/podman/test/e2e/run_test.go:21
  podman run a container without --init [It]
  /home/aidan/foss/podman/test/e2e/run_test.go:216

  Expected
      <int>: 126
  to equal
      <int>: 0

  /home/aidan/foss/podman/test/e2e/run_test.go:219

  Full Stack Trace
  github.com/containers/podman/v2/test/e2e.glob..func86.16()
        /home/aidan/foss/podman/test/e2e/run_test.go:219 +0x1ff
  github.com/onsi/ginkgo/internal/leafnodes.(*runner).runSync(0xc0007c4b40, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/aidan/foss/podman/_output/pkg/mod/github.com/onsi/ginkgo@v1.14.2/internal/leafnodes/runner.go:113 +0xa3
  github.com/onsi/ginkgo/internal/leafnodes.(*runner).run(0xc0007c4b40, 0x17975c8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/aidan/foss/podman/_output/pkg/mod/github.com/onsi/ginkgo@v1.14.2/internal/leafnodes/runner.go:64 +0xcf
  github.com/onsi/ginkgo/internal/leafnodes.(*ItNode).Run(0xc000304340, 0x1995d00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/aidan/foss/podman/_output/pkg/mod/github.com/onsi/ginkgo@v1.14.2/internal/leafnodes/it_node.go:26 +0x64
  github.com/onsi/ginkgo/internal/spec.(*Spec).runSample(0xc0008b1770, 0x0, 0x1995d00, 0xc00004ee40)
        /home/aidan/foss/podman/_output/pkg/mod/github.com/onsi/ginkgo@v1.14.2/internal/spec/spec.go:215 +0x638
  github.com/onsi/ginkgo/internal/spec.(*Spec).Run(0xc0008b1770, 0x1995d00, 0xc00004ee40)
        /home/aidan/foss/podman/_output/pkg/mod/github.com/onsi/ginkgo@v1.14.2/internal/spec/spec.go:138 +0xf2
  github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).runSpec(0xc00098a000, 0xc0008b1770, 0x0)
        /home/aidan/foss/podman/_output/pkg/mod/github.com/onsi/ginkgo@v1.14.2/internal/specrunner/spec_runner.go:200 +0x10f
  github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).runSpecs(0xc00098a000, 0x1)
        /home/aidan/foss/podman/_output/pkg/mod/github.com/onsi/ginkgo@v1.14.2/internal/specrunner/spec_runner.go:170 +0x120
  github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).Run(0xc00098a000, 0xc000322030)
        /home/aidan/foss/podman/_output/pkg/mod/github.com/onsi/ginkgo@v1.14.2/internal/specrunner/spec_runner.go:66 +0x117
  github.com/onsi/ginkgo/internal/suite.(*Suite).Run(0xc0000f5f80, 0x7f9b4e4d11b0, 0xc0006847e0, 0x16ede24, 0xc, 0xc00040f0a0, 0x1, 0x1, 0x19c9aa0, 0xc00004ee40, ...)
        /home/aidan/foss/podman/_output/pkg/mod/github.com/onsi/ginkgo@v1.14.2/internal/suite/suite.go:79 +0x586
  github.com/onsi/ginkgo.RunSpecsWithCustomReporters(0x1996fe0, 0xc0006847e0, 0x16ede24, 0xc, 0xc000079f28, 0x1, 0x1, 0x417528)
        /home/aidan/foss/podman/_output/pkg/mod/github.com/onsi/ginkgo@v1.14.2/ginkgo_dsl.go:229 +0x220
  github.com/onsi/ginkgo.RunSpecs(0x1996fe0, 0xc0006847e0, 0x16ede24, 0xc, 0x1631f59726534)
        /home/aidan/foss/podman/_output/pkg/mod/github.com/onsi/ginkgo@v1.14.2/ginkgo_dsl.go:210 +0x16e
  github.com/containers/podman/v2/test/e2e.TestLibpod(0xc0006847e0)
        /home/aidan/foss/podman/test/e2e/common_test.go:99 +0x102
  testing.tRunner(0xc0006847e0, 0x1793e60)
        /usr/lib/golang/src/testing/testing.go:1050 +0xdc
  created by testing.(*T).Run
        /usr/lib/golang/src/testing/testing.go:1095 +0x28b

Steps to reproduce the issue:

  1. make test

Describe the results you received:

/home/aidan/foss/podman/test/e2e/run_test.go:21
  podman run a container without --init [It]
  /home/aidan/foss/podman/test/e2e/run_test.go:216

  Expected
      <int>: 126
  to equal
      <int>: 0

  /home/aidan/foss/podman/test/e2e/run_test.go:219

  Full Stack Trace
  github.com/containers/podman/v2/test/e2e.glob..func86.16()
        /home/aidan/foss/podman/test/e2e/run_test.go:219 +0x1ff
  github.com/onsi/ginkgo/internal/leafnodes.(*runner).runSync(0xc0007c4b40, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/aidan/foss/podman/_output/pkg/mod/github.com/onsi/ginkgo@v1.14.2/internal/leafnodes/runner.go:113 +0xa3
  github.com/onsi/ginkgo/internal/leafnodes.(*runner).run(0xc0007c4b40, 0x17975c8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/aidan/foss/podman/_output/pkg/mod/github.com/onsi/ginkgo@v1.14.2/internal/leafnodes/runner.go:64 +0xcf
  github.com/onsi/ginkgo/internal/leafnodes.(*ItNode).Run(0xc000304340, 0x1995d00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/aidan/foss/podman/_output/pkg/mod/github.com/onsi/ginkgo@v1.14.2/internal/leafnodes/it_node.go:26 +0x64
  github.com/onsi/ginkgo/internal/spec.(*Spec).runSample(0xc0008b1770, 0x0, 0x1995d00, 0xc00004ee40)
        /home/aidan/foss/podman/_output/pkg/mod/github.com/onsi/ginkgo@v1.14.2/internal/spec/spec.go:215 +0x638
  github.com/onsi/ginkgo/internal/spec.(*Spec).Run(0xc0008b1770, 0x1995d00, 0xc00004ee40)
        /home/aidan/foss/podman/_output/pkg/mod/github.com/onsi/ginkgo@v1.14.2/internal/spec/spec.go:138 +0xf2
  github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).runSpec(0xc00098a000, 0xc0008b1770, 0x0)
        /home/aidan/foss/podman/_output/pkg/mod/github.com/onsi/ginkgo@v1.14.2/internal/specrunner/spec_runner.go:200 +0x10f
  github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).runSpecs(0xc00098a000, 0x1)
        /home/aidan/foss/podman/_output/pkg/mod/github.com/onsi/ginkgo@v1.14.2/internal/specrunner/spec_runner.go:170 +0x120
  github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).Run(0xc00098a000, 0xc000322030)
        /home/aidan/foss/podman/_output/pkg/mod/github.com/onsi/ginkgo@v1.14.2/internal/specrunner/spec_runner.go:66 +0x117
  github.com/onsi/ginkgo/internal/suite.(*Suite).Run(0xc0000f5f80, 0x7f9b4e4d11b0, 0xc0006847e0, 0x16ede24, 0xc, 0xc00040f0a0, 0x1, 0x1, 0x19c9aa0, 0xc00004ee40, ...)
        /home/aidan/foss/podman/_output/pkg/mod/github.com/onsi/ginkgo@v1.14.2/internal/suite/suite.go:79 +0x586
  github.com/onsi/ginkgo.RunSpecsWithCustomReporters(0x1996fe0, 0xc0006847e0, 0x16ede24, 0xc, 0xc000079f28, 0x1, 0x1, 0x417528)
        /home/aidan/foss/podman/_output/pkg/mod/github.com/onsi/ginkgo@v1.14.2/ginkgo_dsl.go:229 +0x220
  github.com/onsi/ginkgo.RunSpecs(0x1996fe0, 0xc0006847e0, 0x16ede24, 0xc, 0x1631f59726534)
        /home/aidan/foss/podman/_output/pkg/mod/github.com/onsi/ginkgo@v1.14.2/ginkgo_dsl.go:210 +0x16e
  github.com/containers/podman/v2/test/e2e.TestLibpod(0xc0006847e0)
        /home/aidan/foss/podman/test/e2e/common_test.go:99 +0x102
  testing.tRunner(0xc0006847e0, 0x1793e60)
        /usr/lib/golang/src/testing/testing.go:1050 +0xdc
  created by testing.(*T).Run
        /usr/lib/golang/src/testing/testing.go:1095 +0x28b

Describe the results you expected:

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

[aidan@unknowne8d0fc9c5665 podman]$ ./bin/podman --version
podman version 2.2.0-dev
[aidan@unknowne8d0fc9c5665 podman]$ git log | head -1
commit ea753128952e1a6d4b56cc80d232f6dbfb420ba5

Output of podman info --debug:

$ ./bin/podman info --debug
host:
  arch: amd64
  buildahVersion: 1.17.0
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.21-2.fc32.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.21, commit: 81d18b6c3ffc266abdef7ca94c1450e669a6a388'
  cpus: 8
  distribution:
    distribution: fedora
    version: "32"
  eventLogger: journald
  hostname: aidan-laptop
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.8.16-200.fc32.x86_64
  linkmode: dynamic
  memFree: 2100625408
  memTotal: 8178700288
  ociRuntime:
    name: runc
    package: containerd.io-1.2.13-3.2.fc31.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc10
      commit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
      spec: 1.0.1-dev
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.4-1.fc32.x86_64
    version: |-
      slirp4netns version 1.1.4
      commit: b66ffa8e262507e37fca689822d23430f3357fe8
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 2
  swapFree: 2526855168
  swapTotal: 8313106432
  uptime: 139h 58m 41.85s (Approximately 5.79 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/aidan/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.2.0-1.fc32.x86_64
      Version: |-
        fusermount3 version: 3.9.1
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.9.1
        using FUSE kernel interface version 7.31
  graphRoot: /home/aidan/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 61
  runRoot: /run/user/1000/containers
  volumePath: /home/aidan/.local/share/containers/storage/volumes
version:
  APIVersion: 2.0.0
  Built: 1605144793
  BuiltTime: Wed Nov 11 17:33:13 2020
  GitCommit: ea753128952e1a6d4b56cc80d232f6dbfb420ba5-dirty
  GoVersion: go1.14.10
  OsArch: linux/amd64
  Version: 2.2.0-dev

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

This is recent master branch build and just trying to run the tests

@openshift-ci-robot openshift-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Nov 12, 2020
@vrothberg
Copy link
Member

Thanks for reaching out! Could you share the entire log? I would expect an error message to appear before the ginkgo log.

@aodhan-domhnaill
Copy link
Author

Logs are attached from sudo make test &> logs.txt.

logs.txt.gz

@rhatdan
Copy link
Member

rhatdan commented Nov 14, 2020

Did you get this to repeat? This is about the most simple container on the system. I would figure this is a hiccup or all of your containers would be blowing up.

@aodhan-domhnaill
Copy link
Author

aodhan-domhnaill commented Nov 14, 2020 via email

@rhatdan
Copy link
Member

rhatdan commented Nov 15, 2020

Podman works with either v1 or v2, not sure if it works with both.

@aodhan-domhnaill
Copy link
Author

Are you able to reproduce this error? I tried

$ ./bin/podman run --rm busybox sleep 5m

# Separate Terminal
$ ./bin/podman exec interesting_tu ls
bin
dev
etc
home
proc
root
run
sys
tmp
usr
var

I'm not sure why specifically the integ tests are failing with 126.

@rhatdan
Copy link
Member

rhatdan commented Nov 16, 2020

This works fine on my F33 box.

The log looks like something with firewall is blowing up on your machine?

Error: error configuring network namespace for container fc4b191e00fb46e97345a253046c4d80df7beccb3a1c46e4bb51589c4b030cfe: failed to add the address 10.88.147.156/32 to trusted zone: COMMAND_FAILED: 'python-nftables' failed:

@aodhan-domhnaill
Copy link
Author

time="2020-11-12T08:33:27-08:00" level=error msg="Error while adding pod to CNI network \"podman\": failed to add the address 10.88.205.147/32 to trusted zone: COMMAND_FAILED: 'python-nftables' failed: \nJSON blob:\n{\"nftables\": [{\"metainfo\": {\"json_schema_version\": 1}}, {\"insert\": {\"rule\": {\"family\": \"inet\", \"table\": \"firewalld\", \"chain\": \"raw_PREROUTING_ZONES\", \"expr\": [{\"match\": {\"left\": {\"payload\": {\"protocol\": \"ip\", \"field\": \"saddr\"}}, \"op\": \"==\", \"right\": {\"prefix\": {\"addr\": \"10.88.205.147\", \"len\": 32}}}}, {\"goto\": {\"target\": \"raw_PRE_trusted\"}}]}}}, {\"insert\": {\"rule\": {\"family\": \"inet\", \"table\": \"firewalld\", \"chain\": \"mangle_PREROUTING_ZONES\", \"expr\": [{\"match\": {\"left\": {\"payload\": {\"protocol\": \"ip\", \"field\": \"saddr\"}}, \"op\": \"==\", \"right\": {\"prefix\": {\"addr\": \"10.88.205.147\", \"len\": 32}}}}, {\"goto\": {\"target\": \"mangle_PRE_trusted\"}}]}}}, {\"insert\": {\"rule\": {\"family\": \"ip\", \"table\": \"firewalld\", \"chain\": \"nat_PREROUTING_ZONES\", \"expr\": [{\"match\": {\"left\": {\"payload\": {\"protocol\": \"ip\", \"field\": \"saddr\"}}, \"op\": \"==\", \"right\": {\"prefix\": {\"addr\": \"10.88.205.147\", \"len\": 32}}}}, {\"goto\": {\"target\": \"nat_PRE_trusted\"}}]}}}, {\"insert\": {\"rule\": {\"family\": \"ip\", \"table\": \"firewalld\", \"chain\": \"nat_POSTROUTING_ZONES\", \"expr\": [{\"match\": {\"left\": {\"payload\": {\"protocol\": \"ip\", \"field\": \"daddr\"}}, \"op\": \"==\", \"right\": {\"prefix\": {\"addr\": \"10.88.205.147\", \"len\": 32}}}}, {\"goto\": {\"target\": \"nat_POST_trusted\"}}]}}}, {\"insert\": {\"rule\": {\"family\": \"inet\", \"table\": \"firewalld\", \"chain\": \"filter_INPUT_ZONES\", \"expr\": [{\"match\": {\"left\": {\"payload\": {\"protocol\": \"ip\", \"field\": \"saddr\"}}, \"op\": \"==\", \"right\": {\"prefix\": {\"addr\": \"10.88.205.147\", \"len\": 32}}}}, {\"goto\": {\"target\": \"filter_IN_trusted\"}}]}}}, {\"insert\": {\"rule\": {\"family\": \"inet\", \"table\": \"firewalld\", \"chain\": \"filter_FORWARD_IN_ZONES\", \"expr\": [{\"match\": {\"left\": {\"payload\": {\"protocol\": \"ip\", \"field\": \"saddr\"}}, \"op\": \"==\", \"right\": {\"prefix\": {\"addr\": \"10.88.205.147\", \"len\": 32}}}}, {\"goto\": {\"target\": \"filter_FWDI_trusted\"}}]}}}, {\"insert\": {\"rule\": {\"family\": \"inet\", \"table\": \"firewalld\", \"chain\": \"filter_FORWARD_OUT_ZONES\", \"expr\": [{\"match\": {\"left\": {\"payload\": {\"protocol\": \"ip\", \"field\": \"daddr\"}}, \"op\": \"==\", \"right\": {\"prefix\": {\"addr\": \"10.88.205.147\", \"len\": 32}}}}, {\"goto\": {\"target\": \"filter_FWDO_trusted\"}}]}}}]}"
Error: error configuring network namespace for container bcdf92d5e44a570d9bc08733f7964a2c08dd285b6a488e09a686a2255f60f27f: failed to add the address 10.88.205.147/32 to trusted zone: COMMAND_FAILED: 'python-nftables' failed:
[aidan@unknowne8d0fc9c5665 podman]$ podman network ls
NAME    VERSION  PLUGINS
[aidan@unknowne8d0fc9c5665 podman]$ sudo podman network ls
NAME    VERSION  PLUGINS
podman  0.4.0    bridge,portmap,firewall,tuning

removing that network doesn't work. I see this error around. Any recommendations to fix it?

@vrothberg
Copy link
Member

Can you do sudo podman system prune -af? This will remove pretty much all Podman data. Do simple commands work?

@aodhan-domhnaill
Copy link
Author

Just tried it. Same problem. I just discovered that podman fails when running under sudo

$ podman run --rm busybox echo hello
hello


$ sudo podman run --rm busybox echo hello
ERRO[0000] Error adding network: failed to add the address 10.88.2.7/32 to trusted zone: COMMAND_FAILED: 'python-nftables' failed: 
JSON blob:
{"nftables": [{"metainfo": {"json_schema_version": 1}}, {"insert": {"rule": {"family": "inet", "table": "firewalld", "chain": "raw_PREROUTING_ZONES", "expr": [{"match": {"left": {"payload": {"protocol": "ip", "field": "saddr"}}, "op": "==", "right": {"prefix": {"addr": "10.88.2.7", "len": 32}}}}, {"goto": {"target": "raw_PRE_trusted"}}]}}}, {"insert": {"rule": {"family": "inet", "table": "firewalld", "chain": "mangle_PREROUTING_ZONES", "expr": [{"match": {"left": {"payload": {"protocol": "ip", "field": "saddr"}}, "op": "==", "right": {"prefix": {"addr": "10.88.2.7", "len": 32}}}}, {"goto": {"target": "mangle_PRE_trusted"}}]}}}, {"insert": {"rule": {"family": "ip", "table": "firewalld", "chain": "nat_PREROUTING_ZONES", "expr": [{"match": {"left": {"payload": {"protocol": "ip", "field": "saddr"}}, "op": "==", "right": {"prefix": {"addr": "10.88.2.7", "len": 32}}}}, {"goto": {"target": "nat_PRE_trusted"}}]}}}, {"insert": {"rule": {"family": "ip", "table": "firewalld", "chain": "nat_POSTROUTING_ZONES", "expr": [{"match": {"left": {"payload": {"protocol": "ip", "field": "daddr"}}, "op": "==", "right": {"prefix": {"addr": "10.88.2.7", "len": 32}}}}, {"goto": {"target": "nat_POST_trusted"}}]}}}, {"insert": {"rule": {"family": "inet", "table": "firewalld", "chain": "filter_INPUT_ZONES", "expr": [{"match": {"left": {"payload": {"protocol": "ip", "field": "saddr"}}, "op": "==", "right": {"prefix": {"addr": "10.88.2.7", "len": 32}}}}, {"goto": {"target": "filter_IN_trusted"}}]}}}, {"insert": {"rule": {"family": "inet", "table": "firewalld", "chain": "filter_FORWARD_IN_ZONES", "expr": [{"match": {"left": {"payload": {"protocol": "ip", "field": "saddr"}}, "op": "==", "right": {"prefix": {"addr": "10.88.2.7", "len": 32}}}}, {"goto": {"target": "filter_FWDI_trusted"}}]}}}, {"insert": {"rule": {"family": "inet", "table": "firewalld", "chain": "filter_FORWARD_OUT_ZONES", "expr": [{"match": {"left": {"payload": {"protocol": "ip", "field": "daddr"}}, "op": "==", "right": {"prefix": {"addr": "10.88.2.7", "len": 32}}}}, {"goto": {"target": "filter_FWDO_trusted"}}]}}}]} 
ERRO[0000] Error while adding pod to CNI network "podman": failed to add the address 10.88.2.7/32 to trusted zone: COMMAND_FAILED: 'python-nftables' failed: 
JSON blob:
{"nftables": [{"metainfo": {"json_schema_version": 1}}, {"insert": {"rule": {"family": "inet", "table": "firewalld", "chain": "raw_PREROUTING_ZONES", "expr": [{"match": {"left": {"payload": {"protocol": "ip", "field": "saddr"}}, "op": "==", "right": {"prefix": {"addr": "10.88.2.7", "len": 32}}}}, {"goto": {"target": "raw_PRE_trusted"}}]}}}, {"insert": {"rule": {"family": "inet", "table": "firewalld", "chain": "mangle_PREROUTING_ZONES", "expr": [{"match": {"left": {"payload": {"protocol": "ip", "field": "saddr"}}, "op": "==", "right": {"prefix": {"addr": "10.88.2.7", "len": 32}}}}, {"goto": {"target": "mangle_PRE_trusted"}}]}}}, {"insert": {"rule": {"family": "ip", "table": "firewalld", "chain": "nat_PREROUTING_ZONES", "expr": [{"match": {"left": {"payload": {"protocol": "ip", "field": "saddr"}}, "op": "==", "right": {"prefix": {"addr": "10.88.2.7", "len": 32}}}}, {"goto": {"target": "nat_PRE_trusted"}}]}}}, {"insert": {"rule": {"family": "ip", "table": "firewalld", "chain": "nat_POSTROUTING_ZONES", "expr": [{"match": {"left": {"payload": {"protocol": "ip", "field": "daddr"}}, "op": "==", "right": {"prefix": {"addr": "10.88.2.7", "len": 32}}}}, {"goto": {"target": "nat_POST_trusted"}}]}}}, {"insert": {"rule": {"family": "inet", "table": "firewalld", "chain": "filter_INPUT_ZONES", "expr": [{"match": {"left": {"payload": {"protocol": "ip", "field": "saddr"}}, "op": "==", "right": {"prefix": {"addr": "10.88.2.7", "len": 32}}}}, {"goto": {"target": "filter_IN_trusted"}}]}}}, {"insert": {"rule": {"family": "inet", "table": "firewalld", "chain": "filter_FORWARD_IN_ZONES", "expr": [{"match": {"left": {"payload": {"protocol": "ip", "field": "saddr"}}, "op": "==", "right": {"prefix": {"addr": "10.88.2.7", "len": 32}}}}, {"goto": {"target": "filter_FWDI_trusted"}}]}}}, {"insert": {"rule": {"family": "inet", "table": "firewalld", "chain": "filter_FORWARD_OUT_ZONES", "expr": [{"match": {"left": {"payload": {"protocol": "ip", "field": "daddr"}}, "op": "==", "right": {"prefix": {"addr": "10.88.2.7", "len": 32}}}}, {"goto": {"target": "filter_FWDO_trusted"}}]}}}]} 
Error: error configuring network namespace for container ccedf1f9b18c67b58843c7b70a6f61bd85cc89fa3cd9bfadea8edc73170e94e8: failed to add the address 10.88.2.7/32 to trusted zone: COMMAND_FAILED: 'python-nftables' failed: 
JSON blob:
{"nftables": [{"metainfo": {"json_schema_version": 1}}, {"insert": {"rule": {"family": "inet", "table": "firewalld", "chain": "raw_PREROUTING_ZONES", "expr": [{"match": {"left": {"payload": {"protocol": "ip", "field": "saddr"}}, "op": "==", "right": {"prefix": {"addr": "10.88.2.7", "len": 32}}}}, {"goto": {"target": "raw_PRE_trusted"}}]}}}, {"insert": {"rule": {"family": "inet", "table": "firewalld", "chain": "mangle_PREROUTING_ZONES", "expr": [{"match": {"left": {"payload": {"protocol": "ip", "field": "saddr"}}, "op": "==", "right": {"prefix": {"addr": "10.88.2.7", "len": 32}}}}, {"goto": {"target": "mangle_PRE_trusted"}}]}}}, {"insert": {"rule": {"family": "ip", "table": "firewalld", "chain": "nat_PREROUTING_ZONES", "expr": [{"match": {"left": {"payload": {"protocol": "ip", "field": "saddr"}}, "op": "==", "right": {"prefix": {"addr": "10.88.2.7", "len": 32}}}}, {"goto": {"target": "nat_PRE_trusted"}}]}}}, {"insert": {"rule": {"family": "ip", "table": "firewalld", "chain": "nat_POSTROUTING_ZONES", "expr": [{"match": {"left": {"payload": {"protocol": "ip", "field": "daddr"}}, "op": "==", "right": {"prefix": {"addr": "10.88.2.7", "len": 32}}}}, {"goto": {"target": "nat_POST_trusted"}}]}}}, {"insert": {"rule": {"family": "inet", "table": "firewalld", "chain": "filter_INPUT_ZONES", "expr": [{"match": {"left": {"payload": {"protocol": "ip", "field": "saddr"}}, "op": "==", "right": {"prefix": {"addr": "10.88.2.7", "len": 32}}}}, {"goto": {"target": "filter_IN_trusted"}}]}}}, {"insert": {"rule": {"family": "inet", "table": "firewalld", "chain": "filter_FORWARD_IN_ZONES", "expr": [{"match": {"left": {"payload": {"protocol": "ip", "field": "saddr"}}, "op": "==", "right": {"prefix": {"addr": "10.88.2.7", "len": 32}}}}, {"goto": {"target": "filter_FWDI_trusted"}}]}}}, {"insert": {"rule": {"family": "inet", "table": "firewalld", "chain": "filter_FORWARD_OUT_ZONES", "expr": [{"match": {"left": {"payload": {"protocol": "ip", "field": "daddr"}}, "op": "==", "right": {"prefix": {"addr": "10.88.2.7", "len": 32}}}}, {"goto": {"target": "filter_FWDO_trusted"}}]}}}]}

@vrothberg
Copy link
Member

I am pretty sure @mheon knows what to do :)

@mheon
Copy link
Member

mheon commented Nov 18, 2020 via email

@aodhan-domhnaill
Copy link
Author

Definitely not what I hoped to hear, but happy to help debug. Ran some of the commands from #5431.

[aidan@aidan-laptop ~]$ sudo firewall-cmd --zone=trusted --list-all
trusted
  target: ACCEPT
  icmp-block-inversion: no
  interfaces: 
  sources: 
  services: 
  ports: 
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 
[aidan@aidan-laptop ~]$ sudo firewall-cmd --reload
Error: COMMAND_FAILED: 'python-nftables' failed: 
JSON blob:
{"nftables": [{"metainfo": {"json_schema_version": 1}}, {"add": {"table": {"family": "inet", "name": "firewalld_policy_drop"}}}, {"add": {"chain": {"family": "inet", "table": "firewalld_policy_drop", "name": "filter_input", "type": "filter", "hook": "input", "prio": 9, "policy": "drop"}}}, {"add": {"chain": {"family": "inet", "table": "firewalld_policy_drop", "name": "filter_forward", "type": "filter", "hook": "forward", "prio": 9, "policy": "drop"}}}, {"add": {"chain": {"family": "inet", "table": "firewalld_policy_drop", "name": "filter_output", "type": "filter", "hook": "output", "prio": 9, "policy": "drop"}}}, {"add": {"rule": {"family": "inet", "table": "firewalld_policy_drop", "chain": "filter_input", "expr": [{"match": {"left": {"ct": {"key": "state"}}, "op": "in", "right": {"set": ["established", "related"]}}}, {"accept": null}]}}}, {"add": {"rule": {"family": "inet", "table": "firewalld_policy_drop", "chain": "filter_forward", "expr": [{"match": {"left": {"ct": {"key": "state"}}, "op": "in", "right": {"set": ["established", "related"]}}}, {"accept": null}]}}}, {"add": {"rule": {"family": "inet", "table": "firewalld_policy_drop", "chain": "filter_output", "expr": [{"match": {"left": {"ct": {"key": "state"}}, "op": "in", "right": {"set": ["established", "related"]}}}, {"accept": null}]}}}]}

Looks like a completely separate issue from podman. Maybe a conflict with docker.

@aodhan-domhnaill
Copy link
Author

I was able to fully reproduce the error on Fedora 33 using podman from the repos (ie. not built from source).

[aidan@aidan-laptop ~]$ sudo dnf install moby-engine
[aidan@aidan-laptop ~]$ sudo firewall-cmd --get-active-zone
docker
  interfaces: docker0
[aidan@aidan-laptop ~]$ sudo podman run --rm busybox echo hello
ERRO[0000] Error adding network: failed to add the address 10.88.2.11/32 to trusted zone: COMMAND_FAILED: 'python-
   ... Removed giant JSON blob
[aidan@aidan-laptop ~]$ sudo dnf remove moby-engine
[aidan@aidan-laptop ~]$ sudo firewall-cmd --get-active-zone
docker
  interfaces: docker0
[aidan@aidan-laptop ~]$ sudo systemctl restart firewalld
[aidan@aidan-laptop ~]$ sudo firewall-cmd --get-active-zone
FedoraWorkstation
  interfaces: wlp2s0
trusted
  interfaces: docker0
[aidan@aidan-laptop ~]$ sudo podman run --rm busybox echo hello
hello

Had to reboot because my wifi became non-functional, but podman still worked after this.

Lesson learned, don't try developing docker competitors with docker installed. Docker will fight back

@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 22, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 22, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Development

No branches or pull requests

5 participants