Support shim v2 e.g. for Kata Containers 2.0

[c3d]

/kind feature

Description

Adding support for the shimv2 interface https://alibaba-cloud.medium.com/cri-and-shimv2-a-new-idea-for-kubernetes-integrating-container-runtime-ab8d22f5716b, useful for example for Kata Containers version 2.0 which no longer supports the earlier CRI interface (see kata-containers/kata-containers#1133 (comment)).

Steps to reproduce the issue:

podman run -it --runtime kata fedora bash

That works with kata being kata-runtime (Kata Containers version 1.0) but not with containerd-shim-kata-v2 (Kata Containers version 2.0).

Output of podman version:

Version:      2.1.1
API Version:  2.0.0
Go Version:   go1.15.2
Built:        Wed Oct  7 18:21:20 2020
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.16.1
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: Unknown
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.11-dev, commit: 77f4a5131bfc344ab4485cd3f1e4644bfa2122e2'
  cpus: 12
  distribution:
    distribution: fedora
    version: "33"
  eventLogger: journald
  hostname: turbo.dinechin.lan
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.9.8-200.fc33.x86_64
  linkmode: dynamic
  memFree: 1521201152
  memTotal: 33522884608
  ociRuntime:
    name: crun
    package: crun-0.15.1-1.fc33.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.15.1
      commit: eb0145e5ad4d8207e84a327248af76663d4e50dd
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  rootless: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 99739828224
  swapTotal: 103846764544
  uptime: 72h 44m 18.82s (Approximately 3.00 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageStore:
    number: 5
  runRoot: /var/run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 2.0.0
  Built: 1602087680
  BuiltTime: Wed Oct  7 18:21:20 2020
  GitCommit: ""
  GoVersion: go1.15.2
  OsArch: linux/amd64
  Version: 2.1.1

Package info (e.g. output of rpm -q podman or apt list podman):

podman-2.1.1-12.fc33.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

No (but I don't expect the latest has the fix, not having seen a mention of "shim v2" in the issues)

Additional environment details (AWS, VirtualBox, physical, etc.):

Physical with Kata

Note

Kata Containers is considering doing the opposite, which is to restore the original CLI commands, see kata-containers/kata-containers#722. However, that would lead to a more inefficient setup (multiple shims), so the shimv2 interface remains interesting in the long run.

[rhatdan]

Interested in opening a PR to fix. I am not sure if this would be easy unless we have a small wrapper OCI that would do the magic for us. Making the changes for podman to connect to a socket, would be a lot more invasive.

[mheon]

Doing this properly would be a pretty significant endeavour - we do have an OCI runtime abstraction but some aspects of our current pattern (e.g. Conmon) are baked deeply into parts of Libpod, including the aforementioned abstraction interface.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[ghost]

BTW, the alternative is to get crio supported in nerdctl.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[EduardoVega]

If this is still wanted and can be added, I would like to give it a try. Based on the comments it seems it is not an easy task, so it might take me some time to understand and get the required changes, but if that's not a problem I'm ok with start doing something here.

[mheon]

If you’re willing to take a shot, go for it! We can probably do this in pieces, migrate support in parts (stuff like attach handling is especially hard to get right, for example)

…

On Sat, May 8, 2021 at 14:00 Eduardo Vega ***@***.***> wrote: If this is still wanted and can be added, I would like to give it a try. Based on the comments it seems it is not an easy task, so it might take me some time to understand and get the required changes, but if that's not a problem I'm ok with start doing something here. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#8579 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB3AOCEPIEVZMP7KHCBOIMDTMV3Z7ANCNFSM4UMDU45Q> .

[EduardoVega]

@mheon understood.

So the idea would be to implement a shimv2 client, so that shims like containerd-shim-kata-v2 can be used with podman.

podman -> OCI wrapper -> shimv2 -> ctrs

In this case conmon wouldn't be part of this, since the shim should take care of the container?

For the initial work, what would you consider can be a good start ? I was thinking to start with a very simple create / start operation

I hope this makes sense.

[mheon]

Sure. This sounds reasonable. It should give us an idea of how complex this could get.

…

On Tue, May 11, 2021 at 20:37 Eduardo Vega ***@***.***> wrote: @mheon <https://github.com/mheon> understood. So the idea would be to implement a shimv2 client, so that shims like containerd-shim-kata-v2 can be used with podman. podman -> OCI wrapper -> shimv2 -> ctrs In this case conmon wouldn't be part of this, since the shim should take care of the container? For the initial work, what would you consider can be a good start ? I was thinking to start with a very simple create / start operation I hope this makes sense. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8579 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB3AOCAO6MZZSAIRAF6RGGDTNHEV5ANCNFSM4UMDU45Q> .

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[EduardoVega]

Still working on this. I've been able to get some progress and now trying to get a minimal attach workflow working (podman run without -d flag)

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@EduardoVega What is going on with this?

[EduardoVega]

@rhatdan I haven't been able to get any progress for the last weeks due to the lack of time, but I should be able to start working on this and finalize what I have done so far for the initial PoC

[rhatdan]

@EduardoVega Any progress?

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[JayDoubleu]

So with katav1 no longer available on fedora i was wondering if it would make sense to bring it back as if this doesn't get implemented we're effectively without any kata containers support for podman.

Would it be reasonable to say that the v1 kata-containers package would work with podman if built manually?

[rhatdan]

It certainly should, or if someone create a katav2 executable which talked to the Kata API and just implemented a wrapper on crun. Since CRUn processes the OCI Runtime, this might be the easiest thing to do.

[rhatdan]

@vrothberg WDYT?

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

I don't think this is a Podman issue, if someone wants to write an executable OCI that talks the kata ship, then Podman would gladly execute it. I don't think there is much value in Podman talking the API directly.

[struanb]

@rhatdan I'd appreciate you explaining your thinking further. Is your thinking that supporting Shim v2 is something Podman should never need to do, or just not now?

After all Podman is "a daemonless container engine for developing, managing, and running OCI Containers on your Linux System."

As I understand it, Shim v2 is designed to be a superior API to v1 for running OCI containers, and this is why the Kata team dropped support for v1.

So, if Podman exists to manage OCI containers, why should it not wish to support Shim v2?

[c3d]

For what it is worth, Kata Containers is a primary application of the shimv2 interface, and the community actively rejected the idea of supporting the CLI. So @dgibson and myself started working on a youki-based wrapper that takes OCI command line as input and calls the shimv2. This is not straightforward, because the shimv2 interface, unlike the command-line interface, is not expected to be invoked from multiple processes.

FWIW, not having podman support has caused the Kata community to lose interest and shift to other tools such as nerdctl for development. To me, it's a shame, because podman is more full-featured and would have allowed us to simplify a number of tasks, notably in the CI.

[rhatdan]

I would fully support youki or crun gaining support. I just don't want to add the bloat to Podman to support this API. When having a separate application that Podman already knows how to execute available does the same thing.

Saying shimv2 does not support multiple processes communicating with it, does not make it have podman with built in support versus launching an OCI cli any better or worse.

[rhatdan]

nerdcll would have the same attributes as an OCI, in that it can come from multiple different processes correct?

[c3d]

I would fully support youki or crun gaining support. I just don't want to add the bloat to Podman to support this API.

I believe that the initial intent of the API (not saying it was realized) was that it would be a simpler replacement, so it would not cause bloat. But I think that in the current state of thing, it's safe to say that you could not get rid of the command-line API.

Saying shimv2 does not support multiple processes communicating with it, does not make it have podman with built in support versus launching an OCI cli any better or worse.

It does (slightly) in cases where there are multiple calls for one operation, e.g. typically at container creation time, or operations that deal with multiple containers (where I think that you could do it with a single API RPC channel open from podman, as opposed to one cli command per container, which would then need to reconnect to the same RPC endpoint repeatedly).

Not having fully implemented the wrapper yet, I cannot fully evaluate the cost of that, though.

[c3d]

nerdcll would have the same attributes as an OCI, in that it can come from multiple different processes correct?

The problem with nerdctl is that AFAICT, it's designed as a debug tool for containerd, i.e. can't be used for CRI-O. So while it may offer a viable solution for Kata Containers developers who only care about testing Kata, it does not help providing a convenient CLI for Kata on Fedora or RHEL with CRI-O or podman.

[struanb]

I'm sharing the following in case it's helpful to anyone who, like us, needed VM isolation for their container workloads, a virtiofs-based solution (ruling out Kata v1) and compatibility with docker run or podman run (ruling out Kata v2).

Due to issues like this one, experienced using Docker/Podman CLI to launch Kata Containers, we built RunCVM (Run Container VM): an experimental open-source Docker container runtime, for launching standard container workloads in VMs.

Please note that RunCVM is not a direct competitor to Kata: as an experimental runtime, RunCVM cannot offer the same levels of stability and support as Kata. However RunCVM may be suitable for some use cases and is compatible with docker run today (with experimental support for podman run). Like Kata v2, RunCVM is also virtiofs-based for speed. RunCVM has minimal system dependencies: it relies on the Linux KVM module, and can even be installed in a GitHub Codespace.

[rhatdan]

Does RunCVM support the same CLI as runc?