[WIP] PoC support for shim v2 to run kata containers v2 #11238

EduardoVega · 2021-08-17T04:02:53Z

This PR is a PoC to integrate the shim v2 as a container monitor to run kata containers version 2. A very basic functionality has been added to create, run and stop containers. Some actions are a bit buggy, mainly those related to attach, interactive and tty but the options can be used.

Fixes #8579

All the code changes are based on https://github.com/cri-o/cri-o/blob/master/internal/oci/runtime_vm.go

An important change to consider is that the version of github.com/golang/protobuf had to be downgraded due to this issue containerd/ttrpc#62

Testing information

Host:

root@default:/home/foo# uname -a
Linux default 4.19.0-16-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19) x86_64 GNU/Linux
root@default:/home/foo# cat /etc/*release*
PRETTY_NAME="Debian GNU/Linux 10 (buster)"

Kata version:

root@default:/home/foo# /opt/kata/bin/kata-runtime version
  : 2.0.3

Examples:

root@default:/home/foo/podman# ./bin/podman run --runtime /opt/kata/bin/containerd-shim-kata-v2 alpine uname -a
time="2021-08-16T22:45:07.152992075-05:00" level=warning msg="sandbox's cgroup won't be updated: cgroup path is empty" name=containerd-shim-v2 pid=929 sandbox=3b208b6c615a source=virtcontainers subsystem=sandbox
Linux 3b208b6c615a 5.4.71 #1 SMP Fri Apr 9 19:03:10 UTC 2021 x86_64 Linux

Bugs:

ctr-c must be pressed to exit the attached mode
podman stop and rm must be run since podman thinks that the container is still running

root@default:/home/foo/podman# ./bin/podman run --runtime /opt/kata/bin/containerd-shim-kata-v2 -it alpine sh
time="2021-08-16T22:49:15.560226053-05:00" level=warning msg="sandbox's cgroup won't be updated: cgroup path is empty" name=containerd-shim-v2 pid=1271 sandbox=8cf1b8d3a0c8 source=virtcontainers subsystem=sandbox
                            / # time="2021-08-16T22:49:15.585753670-05:00" level=error msg="post event" error="failed to publish event: exit status 125" name=containerd-shim-v2 pid=1271 sandbox=8cf1b8d3a0c8 source=containerd-kata-shim-v2
                                                     time="2021-08-16T22:49:15.606020074-05:00" level=error msg="post event" error="failed to publish event: exit status 125" name=containerd-shim-v2 pid=1271 sandbox=8cf1b8d3a0c8 source=containerd-kata-shim-v2

/ # uname -a
Linux 8cf1b8d3a0c8 5.4.71 #1 SMP Fri Apr 9 19:03:10 UTC 2021 x86_64 Linux
/ # hostname
8cf1b8d3a0c8
/ # id
uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video)

Bugs:

when trying to exit the container the terminal gets stuck and returns the error ERRO[0061] Failed to close stdin for container "8cf1b8d3a0c8" error="invalid argument: Invalid exec id: invalid argument"

Signed-off-by: Eduardo Vega edvegavalerio@gmail.com

openshift-ci · 2021-08-17T04:03:16Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: EduardoVega
To complete the pull request process, please assign tomsweeneyredhat after the PR has been reviewed.
You can assign the PR to them by writing /assign @tomsweeneyredhat in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

rhatdan · 2021-08-17T20:42:30Z

@EduardoVega Could you check to see how much the size of Podman grows with this PR?

TomSweeneyRedHat · 2021-08-19T00:01:46Z

I love wee little PRs.
Test system doesn't like it at the moment.

EduardoVega · 2021-08-19T02:13:20Z

@rhatdan

go version go1.15.14 linux/amd64

main

-rwxrwxr-x. 1 edu edu 53M Aug 18 19:53 ./bin/podman

shimv2 branch

-rwxrwxr-x. 1 edu edu 56M Aug 18 20:04 ./bin/podman

@TomSweeneyRedHat

My bad, first time working on something this big, I can definitely make any change to improve this.

rhatdan · 2021-08-20T10:11:21Z

What is the benefit of this over executing kata OCI Runtime directly? Is there any way to do this without sucking in so much code.

mheon · 2021-08-20T13:25:37Z

I think the benefit is that the Kata runtime is deprecated and going away 😄

I'll try and do a full review this afternoon

rhatdan · 2021-08-21T12:18:45Z

Adding three megabytes in size in order to talk to a socket, seems a little extreme.

flouthoc · 2021-08-27T15:52:41Z

libpod/oci_util.go

+func isShimv2(name string, paths []string) bool {
+	// Check if oci runtime name is shimv2
+	// i.e containerd.shim.kata.v2
+	r, _ := regexp.Compile(shimv2NameRegex)


Could we use string contains or match operations did not understand the use-case of regexp it seems costly.

thanks, I'll replace this with a new field in containers.conf to specify when shimv2 is used as suggested in the other comment.

flouthoc · 2021-08-27T15:52:55Z

libpod/oci_util.go

+
+	// Check if any of the oci runtime paths is shimv2
+	// i.e /path/to/containerd-shim-kata-v2
+	r, _ = regexp.Compile(shimv2BinaryRegex)


Same could we try string match or contains instead of regexp

Same here, this will change.

flouthoc · 2021-08-27T15:54:29Z

libpod/runtime.go

@@ -441,7 +441,8 @@ func makeRuntime(ctx context.Context, runtime *Runtime) (retErr error) {
 			runtime.defaultOCIRuntime = ociRuntime
 		}
 	}
-	logrus.Debugf("Using OCI runtime %q", runtime.defaultOCIRuntime.Path())
+	logrus.Debugf("Using OCI runtime name: %q", runtime.defaultOCIRuntime.Name())


Nit: Using OCI runtime %s with path %s instead of two log lines.

flouthoc · 2021-08-27T15:59:23Z

libpod/shimv2.go

+	"syscall"
+	"time"
+
+	tasktypes "github.com/containerd/containerd/api/types/task"


Not sure can we abstract this into podman, vendoring seems too much just for a small use-case. I maybe wrong could you help me understand.

Yup Size matters.

ok, I'll try to abstract this into podman. thanks

rhatdan · 2021-08-27T18:11:33Z

How does Podman monitor this.

podman run -d --runtime=kata ...

How do we get notified that the container exited, and what is the status? No conmon to watch it. How do we cleanup the storage?

mheon · 2021-08-27T18:19:43Z

libpod/oci_util.go

+
+// isShimv2 verifies if the oci runtime needs to use shimv2 daemon
+// to create containers i.e kata containers v2
+func isShimv2(name string, paths []string) bool {


Instead of regexing, it's probably safer to do something in containers.conf indicating this is a shimv2 runtime - explicitly set it in the config file for safety.

yes, that's definitely better. I have seen that this is referred as a runtime_type=vm or runtime_version=v2 , is there any preference for this ?

Should this be also configured as a flag when running podman ? i.e podman run --runtime_type=v2 ...

I would just have it be --runtime, let's not add more flags.

mheon · 2021-08-27T18:20:24Z

libpod/shimv2.go

@@ -0,0 +1,776 @@
+// +build linux


I think we need a dedicated build tag for this - kata, maybe - so we can optional build out support.

mheon · 2021-08-27T18:22:48Z

libpod/shimv2.go

+	return "", fmt.Errorf("not supported")
+}
+
+func (r *shimv2) UpdateContainerStatus(ctr *Container) error {


Is this the only way Kata updates us on containers exiting? IE, We have to poll them to see when containers exited? My primary concerns are round container exit - we don't have a Conmon, so we don't have a cleanup process and the container will not be unmounted into the user manually invokes a Podman command that updates its status.

I agree, Podman has gone away, so something needs to record the status of the container. In my opinion this is still conmon. Just we need a way for conmon to wait for container exit.

From my understanding, the shimv2 daemon when is started, uses a ttrpc server exposed by containerd to publish events back.

I tried to mimic what cri-o was doing since both rely on conmon, and I think this is not being set when kata v2 is used. I will try to understand more about how cri-o handles kata container events

openshift-ci · 2021-09-01T12:54:47Z

@EduardoVega: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

EduardoVega · 2021-10-08T00:55:24Z

Updates

After some more research, I was able to understand better how the Shimv2 can be used with Podman, so the following diagram will try to explain it

                                                          Kata container
                                                         ┌──────────────┐
                                                         │      VM      │
┌───────────────┐           ┌─────────────┐              │              │
│               │  creates  │             │  creates     │ ┌──────────┐ │
│    Podman     ├──────────►│   Shimv2    ├─────────────►│ │          │ │
│               │           │             │              │ │ container│ │
└───────────────┘           └──────┬──────┘              │ │          │ │
                                   │                     │ └──────────┘ │
                                   │publishes events     │              │
                                   │                     └──────────────┘
                                   ▼
                           ┌───────────────┐
                           │               │
                           │    Podman     │
                           │               │
                           └───────────────┘

Podman validates in containers.conf if runtime supports Shimv2
If supported, Podman creates Shimv2 daemon
Shimv2 manages container operations (create, start, stop, ...)
Shimv2 publishes container events to publish-binary (Podman) that is passed as an arg during Shimv2 creation
- Podman will provide a new sub command publish that will receive and use this events to update the state of the container

So based on these findings and the feedback provided in this PR, these could be the following items to be worked

Add a new property in containers.conf to determine which runtimes support the Shimv2 implementation
Add small workflow to run a container with --detach flag, handle events, stop and delete container
- Size matters, if code can be abstracted it will be instead of vendor modules
- Add flag to enable this support during compilation time
Add support for remaining operations and execution modes (--tty, --interactive, ...)
- Small PRs to make review process easier
Add testing for Podman and Shimv2

Finally, this past months I've been very busy but I should be able to get more progress on this now. Thanks

rhatdan · 2021-10-08T09:39:34Z

This sounds good, but we still need to be able to run traditional OCI containers and shimv2 containers at the same time.

podman ps
Needs to show both.

It needs to be easy to switch between the two

podman --runtime shimv2 run ....
podman --runtime crun run ....

Not sure we want to call this shimv2, perhaps katav2 or something like that might be easier for users to understand.

EduardoVega · 2021-10-08T11:32:39Z

@rhatdan understood.

The shimv2 is just an interface that can be implemented by any runtime project, examples:

kata containers v2
https://github.com/firecracker-microvm/firecracker-containerd

This is an example of how this could be run

containers.conf

runtime_supports_shimv2 = ["katav2"]

katav2 = [
             "/opt/kata/bin/containerd-shim-kata-v2"
]

Run Podman

podman run --runtime katav2 ...

Let me know if I'm missing something, thanks.

rhatdan · 2021-10-08T12:29:37Z

No that looks perfect.

This PR is a PoC to integrate the shimv2 as a container runtime to run kata containers version 2. A very basic functionality has been added to create, run and stop containers. Some actions are a bit buggy, mainly those related to attach, interactive and tty but the options can be used. Signed-off-by: Eduardo Vega <edvegavalerio@gmail.com>

github-actions · 2021-12-04T00:04:09Z

A friendly reminder that this PR had no activity for 30 days.

rhatdan · 2021-12-06T11:55:37Z

@EduardoVega Still working on this?

github-actions · 2022-01-06T00:04:46Z

A friendly reminder that this PR had no activity for 30 days.

rhatdan · 2022-01-06T13:48:51Z

Since I have not heard back, I am going to close this PR, if/when someone wants to work on it, we can reopeon or create a new PR.

flouthoc reviewed Aug 27, 2021

View reviewed changes

mheon reviewed Aug 27, 2021

View reviewed changes

openshift-ci bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 1, 2021

EduardoVega changed the title ~~PoC support for shim v2 to run kata containers v2~~ [WIP] PoC support for shim v2 to run kata containers v2 Sep 22, 2021

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 22, 2021

zer0def mentioned this pull request Oct 21, 2021

re-add OCI CLI commands for podman (Note: docker is now supported with docker-23.0) kata-containers/kata-containers#722

Open

EduardoVega force-pushed the 8579-shimv2-integration branch from ddf8f4d to 2b8eeb8 Compare November 3, 2021 21:38

github-actions bot added the stale-pr label Dec 4, 2021

rhatdan removed the stale-pr label Dec 6, 2021

github-actions bot added the stale-pr label Jan 6, 2022

rhatdan closed this Jan 6, 2022

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 22, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 22, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] PoC support for shim v2 to run kata containers v2 #11238

[WIP] PoC support for shim v2 to run kata containers v2 #11238

EduardoVega commented Aug 17, 2021

openshift-ci bot commented Aug 17, 2021

rhatdan commented Aug 17, 2021

TomSweeneyRedHat commented Aug 19, 2021

EduardoVega commented Aug 19, 2021

rhatdan commented Aug 20, 2021

mheon commented Aug 20, 2021

rhatdan commented Aug 21, 2021

flouthoc Aug 27, 2021

EduardoVega Aug 31, 2021

flouthoc Aug 27, 2021 •

edited

Loading

EduardoVega Aug 31, 2021

flouthoc Aug 27, 2021

flouthoc Aug 27, 2021 •

edited

Loading

rhatdan Aug 27, 2021

EduardoVega Aug 31, 2021

rhatdan commented Aug 27, 2021

mheon Aug 27, 2021

EduardoVega Aug 31, 2021

rhatdan Aug 31, 2021

mheon Aug 27, 2021

mheon Aug 27, 2021

rhatdan Aug 27, 2021

EduardoVega Aug 31, 2021 •

edited

Loading

openshift-ci bot commented Sep 1, 2021

EduardoVega commented Oct 8, 2021 •

edited

Loading

rhatdan commented Oct 8, 2021

EduardoVega commented Oct 8, 2021

rhatdan commented Oct 8, 2021

github-actions bot commented Dec 4, 2021

rhatdan commented Dec 6, 2021

github-actions bot commented Jan 6, 2022

rhatdan commented Jan 6, 2022

[WIP] PoC support for shim v2 to run kata containers v2 #11238

[WIP] PoC support for shim v2 to run kata containers v2 #11238

Conversation

EduardoVega commented Aug 17, 2021

openshift-ci bot commented Aug 17, 2021

rhatdan commented Aug 17, 2021

TomSweeneyRedHat commented Aug 19, 2021

EduardoVega commented Aug 19, 2021

rhatdan commented Aug 20, 2021

mheon commented Aug 20, 2021

rhatdan commented Aug 21, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

flouthoc Aug 27, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

flouthoc Aug 27, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rhatdan commented Aug 27, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

EduardoVega Aug 31, 2021 • edited Loading

Choose a reason for hiding this comment

openshift-ci bot commented Sep 1, 2021

EduardoVega commented Oct 8, 2021 • edited Loading

rhatdan commented Oct 8, 2021

EduardoVega commented Oct 8, 2021

rhatdan commented Oct 8, 2021

github-actions bot commented Dec 4, 2021

rhatdan commented Dec 6, 2021

github-actions bot commented Jan 6, 2022

rhatdan commented Jan 6, 2022

flouthoc Aug 27, 2021 •

edited

Loading

flouthoc Aug 27, 2021 •

edited

Loading

EduardoVega Aug 31, 2021 •

edited

Loading

EduardoVega commented Oct 8, 2021 •

edited

Loading