[macos] Error: cannot connect to the Podman socket, please verify that Podman REST API service is running: server API version is too old. Client "3.2.0" server "3.1.2"

[thomasw64]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

When trying to interact with podman on macOS 11.4 (both Apple Silicon M1 as well as x86), after creating and starting the machine with podman machine init and podman machine start I tried to validate the configuration. Any command that goes to the VM comes back with the following message:

Error: cannot connect to the Podman socket, please verify that Podman REST API service is running: server API version is too old. Client "3.2.0" server "3.1.2"

A manual update of the underlying fcos did not work.

Connecting to vm podman-machine-default. To close connection, use `~.` or `exit`


Fedora CoreOS 34.20210601.dev.0
Tracker: https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/c/server/coreos/

Last login: Sun Jun  6 13:23:24 2021 from 10.0.2.2
[core@localhost ~]$ 
[core@localhost ~]$ sudo rpm-ostree upgrade
error: While pulling fedora/aarch64/coreos/testing-devel: No such branch 'fedora/aarch64/coreos/testing-devel' in repository summary
[core@localhost ~]$ sudo rpm-ostree update
error: While pulling fedora/aarch64/coreos/testing-devel: No such branch 'fedora/aarch64/coreos/testing-devel' in repository summary

The URL for the image https://fedorapeople.org/groups/fcos-images/builds/latest/aarch64/fedora-coreos-34.20210601.dev.0-qemu.aarch64.qcow2.xz is set by podman/pkg/machine/fcos_arm64.go

This problem will re-appear every time the API changes, and the updates between macOS podman and fcos are not synchronized.

Steps to reproduce the issue:

1. create a machine and start it. (It might be needed to make a circumvention for [macos] podman machine start on Apple M1 uses qemu with hvf which is not yet supported by qemu on aarch64 #10577 changing to -accel tcg)

podman machine init
podman machine start

2. Try any command that talks to the VM, e.g. search, pull, info, run, ...

#podman pull fedora-minimal:latest     
Error: cannot connect to the Podman socket, please verify that Podman REST API service is running: server API version is too old. Client "3.2.0" server "3.1.2"

Describe the results you received:

Apparently the macOS side runs podman 3.2.0, but the fedora-coros that is started as VM runs still 3.1.2.

Describe the results you expected:

I would have expected that the podman versions in macOS and coreos match.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

from MacOS: podman version

Error: cannot connect to the Podman socket, please verify that Podman REST API service is running: server API version is too old. Client "3.2.0" server "3.1.2"

from inside the coreos

Version:      3.1.2
API Version:  3.1.2
Go Version:   go1.16.3
Built:        Wed May 12 19:28:59 2021
OS/Arch:      linux/arm64

Output of podman info --debug:

host:
  arch: arm64
  buildahVersion: 1.20.1
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.27-2.fc34.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.27, commit: '
  cpus: 1
  distribution:
    distribution: fedora
    version: "34"
  eventLogger: journald
  hostname: localhost
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.12.8-300.fc34.aarch64
  linkmode: dynamic
  memFree: 1633869824
  memTotal: 2059620352
  ociRuntime:
    name: crun
    package: crun-0.19.1-3.fc34.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 0.19.1
      commit: 1535fedf0b83fb898d449f9680000f729ba719f5
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    selinuxEnabled: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.9-1.fc34.aarch64
    version: |-
      slirp4netns version 1.1.8+dev
      commit: 6dc0186e020232ae1a6fcc1f7afbc3ea02fd3876
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 0
  swapTotal: 0
  uptime: 24m 5.77s
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.5.0-1.fc34.aarch64
      Version: |-
        fusermount3 version: 3.10.3
        fuse-overlayfs: version 1.5
        FUSE library version 3.10.3
        using FUSE kernel interface version 7.31
  graphRoot: /var/home/core/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 0
  runRoot: /run/user/1000/containers
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 3.1.2
  Built: 1620847739
  BuiltTime: Wed May 12 19:28:59 2021
  GitCommit: ""
  GoVersion: go1.16.3
  OsArch: linux/arm64
  Version: 3.1.2

Package info (e.g. output of rpm -q podman or apt list podman):

rpm-ostree status

State: idle
Deployments:
● ostree://fedora:fedora/aarch64/coreos/testing-devel
                   Version: 34.20210601.dev.0 (2021-06-01T04:48:54Z)
                    Commit: 5e0e4339273a6335d79cb6937aa3817da7e963c7875a080bb700a7b4ab0d051c
              GPGSignature: (unsigned)

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

MacOs 11.4
on Mac Mini M1 (arm64)
podman 3.2.0 via brew

[thomasw64]

This bug also appears on MacOS / x86. Here is a podman version from an x86 Mac Mini

Connecting to vm podman-machine-default. To close connection, use `~.` or `exit`
The authenticity of host '[localhost]:60357 ([127.0.0.1]:60357)' can't be established.
ECDSA key fingerprint is SHA256:SL+YKZeYGfXVtVyouame9uLd7xluLurmEbAY30x1448.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '[localhost]:60357' (ECDSA) to the list of known hosts.
Fedora CoreOS 34.20210529.1.0
Tracker: https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/c/server/coreos/

[core@localhost ~]$ podman version
Version:      3.1.2
API Version:  3.1.2
Go Version:   go1.16.3
Built:        Wed May 12 19:27:59 2021
OS/Arch:      linux/amd64

and here the failure:

# podman info --debug
Error: cannot connect to the Podman socket, please verify that Podman REST API service is running: server API version is too old. Client "3.2.0" server "3.1.2

[mheon]

I know that M1 support is still pending (need Qemu upstream to merge a few more patches). For the x86 bits, I'll poke @baude

[Jean-Daniel]

At least, you can relax the version check:

By switching to the testing release on Linux I now have the following frustrating error:

server API version is too old. Client "3.2.0" serve "3.2.0-rc3"

Maybe it should be a warning instead of a hard error if major.minor.debug matches.

[mheon]

We're fixing this one in 3.2.1 - this was my mistake, I bumped the minimum supported APi version for the client to 3.2.0, it should be lower.

[kevinGan]

same issue, wait big brother to fix

[thomasw64]

Thank You!!! This is working with 3.2.2. (didn't have the time to test on 3.2.1).

# podman --version
podman version 3.2.2

Here a short trial-run:

# podman run --name fedora --rm fedora-minimal:latest echo 123
Resolved "fedora-minimal" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull registry.fedoraproject.org/fedora-minimal:latest...
Getting image source signatures
Copying blob sha256:554e47f8e1f035f5b6f8b01d03859565998511780f0e511ec1404a51efb1b9de
Copying config sha256:99dc5de598f595acef3a08185d82dd0fbd21b61712653632dffeba114e787bef
Writing manifest to image destination
Storing signatures
123

podman info --debug output:

host:
  arch: arm64
  buildahVersion: 1.20.1
  cgroupControllers: null
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.27-2.fc34.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.27, commit: '
  cpus: 1
  distribution:
    distribution: fedora
    version: "34"
  eventLogger: journald
  hostname: localhost
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.12.8-300.fc34.aarch64
  linkmode: dynamic
  memFree: 1473757184
  memTotal: 2059620352
  ociRuntime:
    name: crun
    package: crun-0.19.1-3.fc34.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 0.19.1
      commit: 1535fedf0b83fb898d449f9680000f729ba719f5
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: ""
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.9-1.fc34.aarch64
    version: |-
      slirp4netns version 1.1.8+dev
      commit: 6dc0186e020232ae1a6fcc1f7afbc3ea02fd3876
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 0
  swapTotal: 0
  uptime: 15m 6.71s
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.5.0-1.fc34.aarch64
      Version: |-
        fusermount3 version: 3.10.3
        fuse-overlayfs: version 1.5
        FUSE library version 3.10.3
        using FUSE kernel interface version 7.31
  graphRoot: /var/home/core/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 3.1.2
  Built: 1620847739
  BuiltTime: Wed May 12 19:28:59 2021
  GitCommit: ""
  GoVersion: go1.16.3
  OsArch: linux/arm64
  Version: 3.1.2