Add new macOS vfkit driver, like hyperkit and qemu #19423
Conversation
k8s-ci-robot
added
cncf-cla: yes
|
Some code is duplicated between pkg/drivers/hyperkit, pkg/drivers/qemu and pkg/drivers/vfkit. It could be considered, to (later!) refactor this into the "common" part under pkg/drivers directly... |
|
This is a very exciting PR ! thank you for thinking about it here is the full log |
|
update: $ time mk start -d vfkit 🔗 Configuring bridge CNI (Container Networking Interface) ... real 0m27.299s I do see one Log spam error though: start.go:132] Unable to get host IP: HostIP not yet implemented for "vfkit" driver |
There was a problem hiding this comment.
I think we need to add a case for cluster/ip.go
func HostIP(host *host.Host, clusterName string) (net.IP, error)
I deployed a service to it and I can hit the minikbe IP directly
vfkit •1
$ mk service hello-minikube
|-----------|----------------|-------------|----------------------------|
| NAMESPACE | NAME | TARGET PORT | URL |
|-----------|----------------|-------------|----------------------------|
| default | hello-minikube | 8080 | http://192.168.106.4:32500 |
|-----------|----------------|-------------|----------------------------|
so I could hit http://192.168.106.4:32500
These are hardcoded on driver name in minikube, so needs another case for vfkit I think it can be copied from hyperkit |
|
/ok-to-test There are no tests for vfkit, but nothing else should have broken either |
k8s-ci-robot
added
the
ok-to-test
This comment has been minimized.
This comment has been minimized.
|
Here are the number of top 10 failed tests in each environments with lowest flake rate.
Besides the following environments also have failed tests:
To see the flake rates of all tests by environment, click here. |
|
Would be great to add functional and integration tests for this in a follow up PR so we can find any bugs and accelerate this out of experimental |
The new experimental vfkit driver[1] uses vfkit to start virtual machines using Apple virtualisation framework when running on Apple silicon. This driver is not available in minikube release yet. To use it you need to install vfkit and build minikube from the PR. Issues: - When terminating the process group, the vfkit process is terminated, and the vm is stopped. This should be fixed in minikube or vfkit by running the vfkit process in background or as a daemon. [1] kubernetes/minikube#19423 [2] https://github.com/crc-org/vfkit Signed-off-by: Nir Soffer <nsoffer@redhat.com>
The new experimental vfkit driver[1] uses vfkit to start virtual
machines using Apple virtualisation framework when running on Apple
silicon.
This driver is not available in minikube release yet. To use it you need
to install vfkit and build minikube from the PR.
Example run:
% drenv start envs/example.yaml
2024-08-16 23:31:18,271 INFO [example] Starting environment
2024-08-16 23:31:18,315 INFO [ex2] Starting minikube cluster
2024-08-16 23:31:18,320 INFO [ex1] Starting minikube cluster
2024-08-16 23:31:42,219 INFO [ex2] Cluster started in 23.90 seconds
2024-08-16 23:31:42,219 INFO [ex2/0] Running addons/example/start
2024-08-16 23:31:49,781 INFO [ex2/0] addons/example/start completed in 7.56 seconds
2024-08-16 23:31:49,781 INFO [ex2/0] Running addons/example/test
2024-08-16 23:31:49,917 INFO [ex2/0] addons/example/test completed in 0.14 seconds
2024-08-16 23:31:50,937 INFO [ex1] Cluster started in 32.62 seconds
2024-08-16 23:31:50,937 INFO [ex1/0] Running addons/example/start
2024-08-16 23:31:58,453 INFO [ex1/0] addons/example/start completed in 7.52 seconds
2024-08-16 23:31:58,453 INFO [ex1/0] Running addons/example/test
2024-08-16 23:31:58,582 INFO [ex1/0] addons/example/test completed in 0.13 seconds
2024-08-16 23:31:58,583 INFO [example] Environment started in 40.31 seconds
% minikube profile list
|---------|-----------|------------|----------------|------|---------|---------|-------|----------------|--------------------|
| Profile | VM Driver | Runtime | IP | Port | Version | Status | Nodes | Active Profile | Active Kubecontext |
|---------|-----------|------------|----------------|------|---------|---------|-------|----------------|--------------------|
| ex1 | vfkit | containerd | 192.168.106.24 | 8443 | v1.31.0 | Running | 1 | | * |
| ex2 | vfkit | containerd | 192.168.106.23 | 8443 | v1.31.0 | Running | 1 | | |
|---------|-----------|------------|----------------|------|---------|---------|-------|----------------|--------------------|
Issues:
- When terminating the process group, the vfkit process is terminated,
and the vm is stopped. This should be fixed in minikube or vfkit by
running the vfkit process in background or as a daemon.
- rook does not work on arm now, likely due to
rook/rook#14502
- We don't have arm64 images for OCM. Can be solved by enabling Rosetta
2. This supported in vfkit but not implemented yet in the vfkit driver.
[1] kubernetes/minikube#19423
[2] https://github.com/crc-org/vfkit
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
The new experimental vfkit driver[1] uses vfkit to start virtual
machines using Apple virtualisation framework when running on Apple
silicon.
This driver is not available in minikube release yet. To use it you need
to install vfkit and build minikube from the PR.
Example run:
% drenv start envs/example.yaml
2024-08-16 23:31:18,271 INFO [example] Starting environment
2024-08-16 23:31:18,315 INFO [ex2] Starting minikube cluster
2024-08-16 23:31:18,320 INFO [ex1] Starting minikube cluster
2024-08-16 23:31:42,219 INFO [ex2] Cluster started in 23.90 seconds
2024-08-16 23:31:42,219 INFO [ex2/0] Running addons/example/start
2024-08-16 23:31:49,781 INFO [ex2/0] addons/example/start completed in 7.56 seconds
2024-08-16 23:31:49,781 INFO [ex2/0] Running addons/example/test
2024-08-16 23:31:49,917 INFO [ex2/0] addons/example/test completed in 0.14 seconds
2024-08-16 23:31:50,937 INFO [ex1] Cluster started in 32.62 seconds
2024-08-16 23:31:50,937 INFO [ex1/0] Running addons/example/start
2024-08-16 23:31:58,453 INFO [ex1/0] addons/example/start completed in 7.52 seconds
2024-08-16 23:31:58,453 INFO [ex1/0] Running addons/example/test
2024-08-16 23:31:58,582 INFO [ex1/0] addons/example/test completed in 0.13 seconds
2024-08-16 23:31:58,583 INFO [example] Environment started in 40.31 seconds
% minikube profile list
|---------|-----------|------------|----------------|------|---------|---------|-------|----------------|--------------------|
| Profile | VM Driver | Runtime | IP | Port | Version | Status | Nodes | Active Profile | Active Kubecontext |
|---------|-----------|------------|----------------|------|---------|---------|-------|----------------|--------------------|
| ex1 | vfkit | containerd | 192.168.106.24 | 8443 | v1.31.0 | Running | 1 | | * |
| ex2 | vfkit | containerd | 192.168.106.23 | 8443 | v1.31.0 | Running | 1 | | |
|---------|-----------|------------|----------------|------|---------|---------|-------|----------------|--------------------|
Issues:
- When terminating the process group, the vfkit process is terminated,
and the vm is stopped. This should be fixed in minikube or vfkit by
running the vfkit process in background or as a daemon.
Fixed in my minikube fork: https://github.com/nirs/minikube/tree/vfkit-pg
- rook does not work on arm now, likely due to
rook/rook#14502
- We don't have arm64 images for OCM. Can be solved by enabling Rosetta
2. This supported in vfkit but not implemented yet in the vfkit driver.
[1] kubernetes/minikube#19423
[2] https://github.com/crc-org/vfkit
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
The new experimental vfkit driver[1] uses vfkit to start virtual
machines using Apple virtualisation framework when running on Apple
silicon.
This driver is not available in minikube release yet. To use it you need
to install vfkit and build minikube from the PR.
Example run:
% drenv start envs/example.yaml
2024-08-16 23:31:18,271 INFO [example] Starting environment
2024-08-16 23:31:18,315 INFO [ex2] Starting minikube cluster
2024-08-16 23:31:18,320 INFO [ex1] Starting minikube cluster
2024-08-16 23:31:42,219 INFO [ex2] Cluster started in 23.90 seconds
2024-08-16 23:31:42,219 INFO [ex2/0] Running addons/example/start
2024-08-16 23:31:49,781 INFO [ex2/0] addons/example/start completed in 7.56 seconds
2024-08-16 23:31:49,781 INFO [ex2/0] Running addons/example/test
2024-08-16 23:31:49,917 INFO [ex2/0] addons/example/test completed in 0.14 seconds
2024-08-16 23:31:50,937 INFO [ex1] Cluster started in 32.62 seconds
2024-08-16 23:31:50,937 INFO [ex1/0] Running addons/example/start
2024-08-16 23:31:58,453 INFO [ex1/0] addons/example/start completed in 7.52 seconds
2024-08-16 23:31:58,453 INFO [ex1/0] Running addons/example/test
2024-08-16 23:31:58,582 INFO [ex1/0] addons/example/test completed in 0.13 seconds
2024-08-16 23:31:58,583 INFO [example] Environment started in 40.31 seconds
% minikube profile list
|---------|-----------|------------|----------------|------|---------|---------|-------|----------------|--------------------|
| Profile | VM Driver | Runtime | IP | Port | Version | Status | Nodes | Active Profile | Active Kubecontext |
|---------|-----------|------------|----------------|------|---------|---------|-------|----------------|--------------------|
| ex1 | vfkit | containerd | 192.168.106.24 | 8443 | v1.31.0 | Running | 1 | | * |
| ex2 | vfkit | containerd | 192.168.106.23 | 8443 | v1.31.0 | Running | 1 | | |
|---------|-----------|------------|----------------|------|---------|---------|-------|----------------|--------------------|
Issues:
- When terminating the process group, the vfkit process is terminated,
and the vm is stopped. This should be fixed in minikube or vfkit by
running the vfkit process in background or as a daemon.
Fixed in my minikube fork: https://github.com/nirs/minikube/tree/vfkit-pg
- rook does not work on arm now, likely due to
rook/rook#14502
- We don't have arm64 images for OCM. Can be solved by enabling Rosetta
2. This supported in vfkit but not implemented yet in the vfkit driver.
- Nested virtualization requires M3 CPU, and is not implemented yet in
vfkit, so testing kubevirt VMs is not possible on Apple silicon.
[1] kubernetes/minikube#19423
[2] https://github.com/crc-org/vfkit
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
There was a problem hiding this comment.
Thanks @afbjorklund, this is awesome!
I'm testing this now with drenv - it works great except one issue with the way the vfkit "daemon" is running.
The missing stuff for our use case is enabling rosetta, since we don't have arm64 images for OCM yet.
| case driver.VFKit: | ||
| vmIPString, _ := host.Driver.GetIP() | ||
| gatewayIPString := vmIPString[:strings.LastIndex(vmIPString, ".")+1] + "1" | ||
| return net.ParseIP(gatewayIPString), nil |
There was a problem hiding this comment.
This duplicates the hyperkit case, which is another implementation of the vmware case.
We can clean this later, but this add more work that nobody will have time to do. Why not do the simple refactoring now, and use the same helper function in all cases when we want to take an ipv4 address and replace the last component with 1?
There was a problem hiding this comment.
This code should really be handled by the driver directly, instead of this switch case poking at driver internals
Notice how hard it is to add extra networking options to the driver, without fiddling around in minikube code
Didn't look into what part of "vanilla" uses HostIP
But using a helper function for hyperkit sounds fair.
There was a problem hiding this comment.
I agree, i would love to see a follow up refactor if someone interested, to move this logic to the driver code not in minikube code
|
|
||
| return nil | ||
| func isBootpdError(err error) bool { |
There was a problem hiding this comment.
isBoopdError() is too string, we don't have a proper way to tell, maybe use something like isLikelyBootpdError() to make our intent more clear.
There was a problem hiding this comment.
Should probably go into common, to be shared with qemu
There was a problem hiding this comment.
agree this should be in the common to be reused and also renamed
| process, err := os.FindProcess(pid) | ||
| if err != nil { | ||
| return err | ||
| } |
There was a problem hiding this comment.
This is racy due to pid reuse, it would be less racy if we get get the process info from the kernel and verify that it is the expected vfkit executable found by exec.Cmd when starting the process.
We can use ps -o comm -p pid, or the same code it uses:
https://github.com/apple-oss-distributions/adv_cmds/blob/8744084ea0ff41ca4bb96b0f9c22407d0e48e9b7/ps/ps.c#L595
There is a Go implementation here:
https://github.com/mitchellh/go-ps/blob/master/process_darwin.go
We can also do this later, it is needed also for other drivers in macOS.
There was a problem hiding this comment.
crc also has some code doing this, https://github.com/crc-org/crc/blob/7b042f76b82b854aabe3e25bc2831ffab987eedb/pkg/drivers/vfkit/driver_darwin.go#L397-L435
There was a problem hiding this comment.
The -pidfile seems simpler
| return state.Error, err | ||
| } | ||
| if err := checkPid(pid); err != nil { | ||
| // No pid, remove pidfile |
There was a problem hiding this comment.
// No process with pid, remove pidfile
There was a problem hiding this comment.
This probably came straight from qemu
| startCmd = append(startCmd, | ||
| "--memory", fmt.Sprintf("%d", d.Memory), | ||
| "--cpus", fmt.Sprintf("%d", d.CPU), | ||
| "--restful-uri", fmt.Sprintf("unix://%s", d.sockfilePath())) |
There was a problem hiding this comment.
Add blank line for consistent whitespace.
| } | ||
|
|
||
| startCmd = append(startCmd, | ||
| "--device", fmt.Sprintf("virtio-blk,path=%s", d.diskPath())) |
There was a problem hiding this comment.
Order of disks matter? If not why not add first the main disk, and then the extra disks?
There was a problem hiding this comment.
Most likely the order was just copied from another driver, whether it was hyperkit or qemu
| pid := cmd.Process.Pid | ||
| if err := os.WriteFile(d.pidfilePath(), []byte(fmt.Sprintf("%v", pid)), 0600); err != nil { | ||
| return err | ||
| } |
There was a problem hiding this comment.
Creating the pid file for vfkit is a bit racy, if we are killed after starting vfkit but before we create the pid file, we will never terminate vfkit.
I think we need to move this vfkit - it should provide a --pidfile option. It goes with the --daemonize option we also need.
There was a problem hiding this comment.
Another race which can happen is that vfkit can exit after cmd.Start() succeeds, for example if Apple's virtualization framework fails to start the VM and returns an error. If this happens, you'd have a pid file but no corresponding process.
I added this code in vfkit's test suite to catch this crc-org/vfkit@a38d84c
It probably would be useful to have a way to tell vfkit "do not exit by yourself, I'll explicitly stop/kill/.. you when I no longer need your process to be around"
The new experimental vfkit driver[1] uses vfkit to start virtual
machines using Apple virtualisation framework when running on Apple
silicon.
This driver is not available in minikube release yet. To use it you need
to install vfkit and build minikube from the PR.
Example run:
% drenv start envs/example.yaml
2024-08-16 23:31:18,271 INFO [example] Starting environment
2024-08-16 23:31:18,315 INFO [ex2] Starting minikube cluster
2024-08-16 23:31:18,320 INFO [ex1] Starting minikube cluster
2024-08-16 23:31:42,219 INFO [ex2] Cluster started in 23.90 seconds
2024-08-16 23:31:42,219 INFO [ex2/0] Running addons/example/start
2024-08-16 23:31:49,781 INFO [ex2/0] addons/example/start completed in 7.56 seconds
2024-08-16 23:31:49,781 INFO [ex2/0] Running addons/example/test
2024-08-16 23:31:49,917 INFO [ex2/0] addons/example/test completed in 0.14 seconds
2024-08-16 23:31:50,937 INFO [ex1] Cluster started in 32.62 seconds
2024-08-16 23:31:50,937 INFO [ex1/0] Running addons/example/start
2024-08-16 23:31:58,453 INFO [ex1/0] addons/example/start completed in 7.52 seconds
2024-08-16 23:31:58,453 INFO [ex1/0] Running addons/example/test
2024-08-16 23:31:58,582 INFO [ex1/0] addons/example/test completed in 0.13 seconds
2024-08-16 23:31:58,583 INFO [example] Environment started in 40.31 seconds
% minikube profile list
|---------|-----------|------------|----------------|------|---------|---------|-------|----------------|--------------------|
| Profile | VM Driver | Runtime | IP | Port | Version | Status | Nodes | Active Profile | Active Kubecontext |
|---------|-----------|------------|----------------|------|---------|---------|-------|----------------|--------------------|
| ex1 | vfkit | containerd | 192.168.106.24 | 8443 | v1.31.0 | Running | 1 | | * |
| ex2 | vfkit | containerd | 192.168.106.23 | 8443 | v1.31.0 | Running | 1 | | |
|---------|-----------|------------|----------------|------|---------|---------|-------|----------------|--------------------|
Issues:
- When terminating the process group, the vfkit process is terminated,
and the vm is stopped. This should be fixed in minikube or vfkit by
running the vfkit process in background or as a daemon.
Fixed in my minikube fork: https://github.com/nirs/minikube/tree/vfkit-pg
- rook does not work on arm now, likely due to
rook/rook#14502
- We don't have arm64 images for OCM. Can be solved by enabling Rosetta
2. This supported in vfkit but not implemented yet in the vfkit driver.
- Nested virtualization requires M3 CPU, and is not implemented yet in
vfkit, so testing kubevirt VMs is not possible on Apple silicon.
[1] kubernetes/minikube#19423
[2] https://github.com/crc-org/vfkit
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
|
I would like the refactoring of the hyperkit driver, including the iso "ExtractFile", to go in a separate PR. But that can be done first, if you don't want the technical debt of simply copying the files to this driver. It (extracting kernel) is only needed because the new vfkit "bootloader" didn't work with minikube.iso (?) With the EFI bootloader, it is supposed to be able to read /boot/bzimage and /boot/initrd (on macOS 13) The pidfile handling was mostly a quick hack when the Those features can be driven to the CRC organization which is handling the "vfkit" (not Podman team, actually) https://github.com/crc-org/vfkit/issues It is mostly developed by one developer |
|
The driver probably needs a "Network" setting, in preparation for future options such as vmnet or hyperkit has the (network is always "vmnet", since it doesn't do cross-platform - like qemu does) qemu has the (network is user or socket, as in https://wiki.qemu.org/Documentation/Networking) vfkit options: https://github.com/crc-org/vfkit/blob/main/doc/usage.md#networking |
This comment has been minimized.
This comment has been minimized.
|
Here are the number of top 10 failed tests in each environments with lowest flake rate.
Besides the following environments also have failed tests:
To see the flake rates of all tests by environment, click here. |
| pathInIso string | ||
| destPath string | ||
| }{ | ||
| {"/boot/bzimage", "bzimage"}, |
There was a problem hiding this comment.
On arm64, vfkit requires an uncompressed kernel, I don't know what format bzimage is in.
Also be aware that there were some issues with direct kernel boot both on amd64 and arm64 on macOS 12 crc-org/vfkit#11
If you can require macOS 13+, I'd recommend using uefi boot if possible, no need to extract kernel/initrd/...
There was a problem hiding this comment.
For some reason it worked fine with boot2docker.iso, but failed to boot with minikube.iso
It might be the extra copy of the kernel, so might require some ISO cleanup before it works?
i.e. there is a BR2_LINUX_KERNEL_INSTALL_TARGET=y that was copied from the BIOS support
Dropping support for macOS 12 is not so much of an issue, I think? Had to upgrade anyway
There was a problem hiding this comment.
The kernel was supposed to be compressed with lz4 (at least so I thought), but seems uncompressed?
BR2_LINUX_KERNEL_LZ4=y
Could be some old bug with the aarch64 iso build perhaps, when it is copying the wrong file or something
Not sure what compression formats that Apple's EFI supports? Probably not lz4, but maybe zstd ?
There was a problem hiding this comment.
Also be aware that there were some issues with direct kernel boot both on amd64 and arm64 on macOS 12 crc-org/vfkit#11
Minikube still uses kernel 5.10, so that should be OK. Perhaps going to upgrade to 5.15, but still (< 5.18)
There was a problem hiding this comment.
I think for new code we can assume current macOS version as requirement. The only reason to run older macOS is old hardware that cannot be upgraded (6+ years old), and such hardware is typically not usable for running minikube with virtual machines.
There was a problem hiding this comment.
The technical requirement is only macOS 11, but there's still the older drivers if you can't use macOS 13
| pid := cmd.Process.Pid | ||
| if err := os.WriteFile(d.pidfilePath(), []byte(fmt.Sprintf("%v", pid)), 0600); err != nil { | ||
| return err | ||
| } |
There was a problem hiding this comment.
Another race which can happen is that vfkit can exit after cmd.Start() succeeds, for example if Apple's virtualization framework fails to start the VM and returns an error. If this happens, you'd have a pid file but no corresponding process.
I added this code in vfkit's test suite to catch this crc-org/vfkit@a38d84c
It probably would be useful to have a way to tell vfkit "do not exit by yourself, I'll explicitly stop/kill/.. you when I no longer need your process to be around"
|
|
||
| // Make a boot2docker VM disk image. | ||
| func (d *Driver) generateDiskImage(size int) error { | ||
| log.Debugf("Creating %d MB hard disk image...", size) |
There was a problem hiding this comment.
Nit: the generated image will be size MiB, not MB.
There was a problem hiding this comment.
It wasn't so important back then (but it was always MiB)
Hopefully this can be decoupled at some future point...
https://github.com/minikube-machine/machine/blob/main/libmachine/mcnutils/b2d.go#L492
I think it's only doing some qemu-img convert / truncate?
There was a problem hiding this comment.
e.g.
// ConvertMBToBytes converts MB to bytes
func ConvertMBToBytes(mbSize int) int64 {
return int64(mbSize) * units.MiB
}There was a problem hiding this comment.
Would also be nice to avoid github.com/docker/go-units
Added the standard "hack" integration script, not sure if there is anything more that needs to be done for testing? maybe |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
Here are the number of top 10 failed tests in each environments with lowest flake rate.
Besides the following environments also have failed tests:
To see the flake rates of all tests by environment, click here. |
|
@afbjorklund #19468 is merged and could be pull into this PR |
It uses the new Virtualization.framework from macOS 11, instead of the older Hypervisor.framework (hvf) in QEMU.
This comment has been minimized.
This comment has been minimized.
| ## Overview | ||
|
|
||
| [VFKit](https://github.com/crc-org/vfkit) is an open-source program for macOS virtualization, optimized for lightweight virtual machines and container deployment. | ||
|
|
There was a problem hiding this comment.
To stay in line with the rest of the driver pages would be good to have a Usage section. I tried to do a suggestion comment but the closing backticks prevent it.
### Usage
```shell
minikube start --driver qemu --network socket_vmnet
Yeah, what you have should be good for integration, might be good to add functional tests as well, but could be done in a follow up PR. |
Co-authored-by: Steven Powell <44844360+spowelljr@users.noreply.github.com>
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: afbjorklund, medyagh The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
the
approved
|
kvm2 driver with docker runtime Times for minikube (PR 19423) start: 51.6s 53.9s 49.8s 51.7s 52.1s Times for minikube ingress: 16.1s 15.5s 16.0s 15.0s 18.5s docker driver with docker runtime Times for minikube (PR 19423) start: 20.7s 24.1s 24.2s 24.0s 20.9s Times for minikube (PR 19423) ingress: 12.8s 10.3s 13.3s 12.8s 10.8s docker driver with containerd runtime Times for minikube start: 20.7s 20.6s 22.3s 22.8s 22.5s Times for minikube ingress: 38.8s 39.8s 38.8s 38.8s 38.9s |
|
This driver works beautifully for running disconnected clusters. The problem is how to make it work with shared network like socket_vmnet, or maybe getting minikube or vfkit a apple.netwroking entitlement, like UTM has, so it can use bridged network. I'm not sure if UTM built locally can use bridged network, or only UTM from Apple app store. But it would be awesome if we can get minikube or vfkit in the app store so it can use good networking - without adding any code. |
It uses the new Virtualization.framework from macOS 11, instead of the older Hypervisor.framework (hvf) in QEMU.
Closes #12826
The "hyperkit" binary was bundled with Docker for Mac, but is available stand-alone:
https://github.com/moby/hyperkit
The "vfkit" binary is bundled with Podman Desktop (macOS), and is available stand-alone:
https://github.com/crc-org/vfkit