Can't use Nested Virtualization in UTM VM

[tr1kn1]

I am using the macOS version of UTM only, so i cannot speak for iOS. It is an Intel Mac running Monterey and the VM is Windows 10 x86_64.

I would like to use some virtualization features inside the guest, but setting CPU flags or changing the CPU model does not change anything: Windows 10 always shows QEMU Virtual CPU version 2.5+.

Is this an accelerator (hvf) issue or could you make this working?

[conath]

Interesting. This reminds me of a recent code change. The -cpu host parameter when no CPU is chosen was removed in a recent update because it caused BSOD specifically in Windows 10 guests - see #2368. However in the new code it seems like the custom CPU model should be passed anyway, if selected in the configuration.

UTM/Managers/UTMQemuSystem.m

Lines 237 to 244 in 6524c73

	if ([self.configuration.systemCPU isEqualToString:@"default"] && self.useHypervisor) {
	// if default and not hypervisor, we don't pass any -cpu argument for x86 and use host for ARM
	#if !defined(__x86_64__)
	[self pushArgv:@"-cpu"];
	[self pushArgv:@"host"];
	#endif
	} else if (self.configuration.systemCPU.length > 0 && ![self.configuration.systemCPU isEqualToString:@"default"]) {
	NSString *cpu = self.configuration.systemCPU;

Please add your VM configuration PLIST so I can examine it.

By the way, is it only the CPU model name in the guest that is not changing to your setting or do the actual feature bits not update when you set them in the UI?

[tr1kn1]

Thanks for your reply!

My config.plist for Windows 10:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>ConfigurationVersion</key>
	<integer>2</integer>
	<key>Debug</key>
	<dict/>
	<key>Display</key>
	<dict>
		<key>ConsoleFont</key>
		<string>Menlo</string>
		<key>ConsoleFontSize</key>
		<integer>12</integer>
		<key>ConsoleOnly</key>
		<false/>
		<key>ConsoleTheme</key>
		<string>Default</string>
		<key>DisplayCard</key>
		<string>qxl-vga</string>
		<key>DisplayDownscaler</key>
		<string>linear</string>
		<key>DisplayFitScreen</key>
		<true/>
		<key>DisplayRetina</key>
		<true/>
		<key>DisplayUpscaler</key>
		<string>linear</string>
	</dict>
	<key>Drives</key>
	<array>
		<dict>
			<key>DriveName</key>
			<string>drive0</string>
			<key>ImagePath</key>
			<string>disk-0.qcow2</string>
			<key>ImageType</key>
			<string>disk</string>
			<key>InterfaceType</key>
			<string>virtio</string>
		</dict>
		<dict>
			<key>DriveName</key>
			<string>drive1</string>
			<key>ImageType</key>
			<string>cd</string>
			<key>InterfaceType</key>
			<string>ide</string>
			<key>Removable</key>
			<true/>
		</dict>
		<dict>
			<key>DriveName</key>
			<string>5A9C8550-FFB6-4F7B-870B-95948D508E6A</string>
			<key>ImagePath</key>
			<string>efi_vars.fd</string>
			<key>ImageType</key>
			<string>none</string>
			<key>InterfaceType</key>
			<string></string>
		</dict>
	</array>
	<key>Info</key>
	<dict>
		<key>Icon</key>
		<string>windows</string>
		<key>IconCustom</key>
		<false/>
	</dict>
	<key>Input</key>
	<dict>
		<key>InputInvertScroll</key>
		<true/>
		<key>InputLegacy</key>
		<false/>
	</dict>
	<key>Networking</key>
	<dict>
		<key>IsolateGuest</key>
		<false/>
		<key>NetworkCard</key>
		<string>virtio-net-pci</string>
		<key>NetworkCardMAC</key>
		<string>$MAC:ADDRESS</string>
		<key>NetworkMode</key>
		<string>emulated</string>
	</dict>
	<key>Printing</key>
	<dict/>
	<key>Sharing</key>
	<dict>
		<key>ClipboardSharing</key>
		<true/>
		<key>DirectoryReadOnly</key>
		<true/>
		<key>DirectorySharing</key>
		<true/>
		<key>Usb3Support</key>
		<true/>
		<key>UsbRedirectMax</key>
		<integer>3</integer>
	</dict>
	<key>Sound</key>
	<dict>
		<key>SoundCard</key>
		<string>intel-hda</string>
		<key>SoundEnabled</key>
		<true/>
	</dict>
	<key>System</key>
	<dict>
		<key>Architecture</key>
		<string>x86_64</string>
		<key>BootDevice</key>
		<string></string>
		<key>BootUefi</key>
		<true/>
		<key>CPU</key>
		<string>max</string>
		<key>CPUCount</key>
		<integer>4</integer>
		<key>CPUFlags</key>
		<array>
			<string>vmx</string>
			<string>sse</string>
			<string>avx</string>
			<string>acpi</string>
		</array>
		<key>ForceMulticore</key>
		<true/>
		<key>MachineProperties</key>
		<string>vmport=off</string>
		<key>Memory</key>
		<integer>8192</integer>
		<key>SystemUUID</key>
		<string>$UUID-String</string>
		<key>Target</key>
		<string>q35</string>
	</dict>
</dict>
</plist>

I must admit that I do not know, if the bits are really set. The behavior inside the VM is not changed, because I cannot activate the WSL features. But CPU-Z tells me, that some features are activated: MMX, SSE, EM64T, AES, AVX, FMA3 for example.

[conath]

You have selected "max" as CPU, please change to either "host" or "default", or a specific CPU model if you really need that (you likely don't). The CPU flags are saved correctly, so no further problems there. I'm wondering if it is indeed to do with the hypervisor.

[conath]

@agraf Does QEMU HVF on Intel Macs support nested virtualization? Thanks.

[agraf]

HVF does not support nested virtualization on x86 (software limitation) as well as M1 (hardware limitation) as far as I know.

[conath]

@osy Do you know if it's possible to use nested virtualization in Apple Silicon VMs (Monterey)?

[agraf]

The M1 hardware does not support nested virtualization. M2 might, but we'll have to wait for it to appear before we know for sure. And even then, there will need to be HVF support in addition to the hardware capability.

[VariableDeclared]

hey @agraf have you got a whitepaper to point to for this? I was looking at AArch64s features, M1 is based on 8.5A, 8.4A supported nested virt? I'm just wondering if this limitation was in software rather than something Apple put in hardware.

[agraf]

The spec does not make every possible feature mandatory. Apple decided to not implement NV1 and NV2.

[VariableDeclared]

okay thanks for that @agraf

[summer-father]

I need to use nested fetrue in utm in the host macOS M2

[redesigned]

Does this mean that we cannot run Docker in a MacOS VM? 😢

[gedw99]

The Apple M1 CPU does not support hardware assisted nested virtualisation at all.

The Apple M2 and M3 CPUs support the v8.4-A level of hardware assisted nested virtualisation.

[kenji21]

It is coming with macOS 15 : https://developer.apple.com/documentation/virtualization/vzgenericplatformconfiguration/4360552-nestedvirtualizationenabled?changes=latest_minor&language=objc

[gedw99]

This will allow to run firecracker

[unizhu]

It is coming with macOS 15 : https://developer.apple.com/documentation/virtualization/vzgenericplatformconfiguration/4360552-nestedvirtualizationenabled?changes=latest_minor&language=objc

So we are waiting for macOS 15 to enable docker in the VM, right?

[unizhu]

hmm.. after upgrading to macOS 15 (both host and VM) , still cannot run docker.. hypervisor check failed. nestedVirtualizationEnabled should be enabled by UTM ...