Upgrade to VirtualBox 7.0 breaks "bosh create-env"

[nicregez]

Summary

I have been using bosh with VirtualBox 6.1.38r153438 for some time now, very happy with it.
Yesterday, I upgraded (unintentionally...) to VirtualBox 7.0.2r154219, now the command bosh create-env fails, see details below.
I uninstalled VirtualBox and reverted back to version 6.1.40r154048, the newest version in the 6.1 line. bosh create-env succeeds again, everything back to normal.

Please let me know how I could contribute to the resolution of this issue.

Environment

• Mac OS 12.6 Intel 32GB RAM.
• Bosh CLI version 7.0.1-799538d1-2022-06-01T21:53:22Z

Steps to reproduce using VirtualBox 7.0

• Execute steps 1 to 4 in https://bosh.io/docs/bosh-lite/ in a brand new VirtualBox setup.
• Step 4 fails

Observation using VirtualBox 7.0

$ bosh create-env ~/workspace/bosh-deployment/bosh.yml \
  --state ./state.json \
  -o ~/workspace/bosh-deployment/virtualbox/cpi.yml \
  -o ~/workspace/bosh-deployment/virtualbox/outbound-network.yml \
  -o ~/workspace/bosh-deployment/bosh-lite.yml \
  -o ~/workspace/bosh-deployment/bosh-lite-runc.yml \
  -o ~/workspace/bosh-deployment/uaa.yml \
  -o ~/workspace/bosh-deployment/credhub.yml \
  -o ~/workspace/bosh-deployment/jumpbox-user.yml \
  --vars-store ./creds.yml \
  -v director_name=bosh-lite \
  -v internal_ip=192.168.56.6 \
  -v internal_gw=192.168.56.1 \
  -v internal_cidr=192.168.56.0/24 \
  -v outbound_network_name=NatNetwork
Deployment manifest: '~/workspace/bosh-deployment/bosh.yml'
Deployment state: './state.json'

Started validating
	  Downloading release 'bosh'... Skipped [Found in local cache] (00:00:00)
  Validating release 'bosh'... Finished (00:00:00)
  Downloading release 'bpm'... Skipped [Found in local cache] (00:00:00)
  Validating release 'bpm'... Finished (00:00:00)
  Downloading release 'bosh-virtualbox-cpi'... Skipped [Found in local cache] (00:00:00)
  Validating release 'bosh-virtualbox-cpi'... Finished (00:00:01)
  Downloading release 'garden-runc'... Skipped [Found in local cache] (00:00:00)
  Validating release 'garden-runc'... Finished (00:00:00)
  Downloading release 'bosh-warden-cpi'... Skipped [Found in local cache] (00:00:00)
  Validating release 'bosh-warden-cpi'... Finished (00:00:00)
  Downloading release 'os-conf'... Skipped [Found in local cache] (00:00:00)
  Validating release 'os-conf'... Finished (00:00:00)
  Downloading release 'uaa'... Skipped [Found in local cache] (00:00:00)
  Validating release 'uaa'... Finished (00:00:00)
  Downloading release 'credhub'... Skipped [Found in local cache] (00:00:00)
  Validating release 'credhub'... Finished (00:00:00)
  Validating cpi release... Finished (00:00:00)
  Validating deployment manifest... Finished (00:00:00)
  Downloading stemcell... Skipped [Found in local cache] (00:00:00)
  Validating stemcell... Finished (00:00:00)
Finished validating (00:00:10)

Started installing CPI
  Compiling package 'golang-1-linux/79f531850e62e3801f1dfa4acd11c421aebe653cd4316f6e49061818071bb617'... Finished (00:00:54)
  Compiling package 'golang-1-darwin/42353203720fdc244cc16f881167bb0363f38ff6f991247c4356e38bc6b5fb81'... Finished (00:01:01)
  Compiling package 'virtualbox_cpi/10a941d387c86d06f0eb3c82f29fb1228489782ba44f8961d3acd264bf2d5198'... Finished (00:00:45)
  Installing packages... Finished (00:00:29)
  Rendering job templates... Finished (00:00:01)
  Installing job 'virtualbox_cpi'... Finished (00:00:00)
Finished installing CPI (00:03:11)

Uploading stemcell 'bosh-vsphere-esxi-ubuntu-jammy-go_agent/1.44'... Finished (00:00:15)

Started deploying
  Creating VM for instance 'bosh/0' from stemcell 'sc-c84c5a5c-536f-4e8b-4eb0-1da712760238'... Failed (00:01:01)
Failed deploying (00:01:01)

Cleaning up rendered CPI jobs... Finished (00:00:00)

Deploying:
  Creating instance 'bosh/0':
    Creating VM:
      Creating vm with stemcell cid 'sc-c84c5a5c-536f-4e8b-4eb0-1da712760238':
        CPI 'create_vm' method responded with error: CmdError{"type":"Bosh::Clouds::CloudError","message":"Creating VM with agent ID '{{bad6d6e8-6e68-4a2d-7161-012cdb2eb4da}}': Enabling networks: Expected to find network 'NatNetwork'","ok_to_retry":false}

Exit code 1

[rkoster]

Have you tried running with export BOSH_LOG_LEVEL=debug? Did it create the NatNetwork?

[jfierro]

The issue with bosh-virtualbox-cpi not finding NatNetwork seems to be due to VBoxManage changing the output format of the list natnetworks command. On version 7 list natnetworks no longer has the Enabled key and the NetworkName and DHCP Enabled keys have been renamed to Name and DHCP Server, respectively. It seems you now have to run natnetwork list to get the same information:

$ VBoxManage natnetwork list
NAT Networks:

Name:         NatNetwork
Network:      10.0.2.0/24
Gateway:      10.0.2.1
DHCP Server:  Yes
IPv6:         No
IPv6 Prefix:  fd17:625c:f037:2::/64
IPv6 Default: No
Enabled:      Yes

1 network found

I got bosh-cli to get past the NatNetwork issue by making a small modification to the bosh-virtualbox-cpi/vm/network/networks.go file (see
networks.zip).

However, it was still failing after that point with this error:

[CLI] 2023/04/05 00:09:19 ERROR - Deploying: Creating instance 'bosh/0': Creating VM: Creating vm with stemcell cid 'sc-c5debf68-2efd-4e98-4efc-457f9f0138fd': CPI 'create_vm' method responded with error: CmdErro
r{"type":"Bosh::Clouds::CloudError","message":"Creating VM with agent ID '{{70164596-b5e9-4771-7e55-0969d9aa5b1c}}': Starting VM: Retried '30' times: Running command: 'VBoxManage startvm vm-0dfbf69b-ed03-4961-73
29-a4cb816b3d9d --type headless', stdout: 'Waiting for VM \"vm-0dfbf69b-ed03-4961-7329-a4cb816b3d9d\" to power on...\n', stderr: 'VBoxManage: error: The virtual machine 'vm-0dfbf69b-ed03-4961-7329-a4cb816b3d9d'
has terminated unexpectedly during startup with exit code 1 (0x1)\nVBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component MachineWrap, interface IMachine\n': exit status 1","ok_to_retry":false
}

Deploying:
  Creating instance 'bosh/0':
    Creating VM:
      Creating vm with stemcell cid 'sc-c5debf68-2efd-4e98-4efc-457f9f0138fd':
        CPI 'create_vm' method responded with error: CmdError{"type":"Bosh::Clouds::CloudError","message":"Creating VM with agent ID '{{70164596-b5e9-4771-7e55-0969d9aa5b1c}}': Starting VM: Retried '30' times: R
unning command: 'VBoxManage startvm vm-0dfbf69b-ed03-4961-7329-a4cb816b3d9d --type headless', stdout: 'Waiting for VM \"vm-0dfbf69b-ed03-4961-7329-a4cb816b3d9d\" to power on...\n', stderr: 'VBoxManage: error: Th
e virtual machine 'vm-0dfbf69b-ed03-4961-7329-a4cb816b3d9d' has terminated unexpectedly during startup with exit code 1 (0x1)\nVBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component MachineWra
p, interface IMachine\n': exit status 1","ok_to_retry":false}

Exit code 1

Anyway, I just downgraded to VirtualBox 6.1.38 and installed the extension pack and it worked seamlessly. Might want to advertise that VirtualBox 7 is not supported yet in the documentation.

[ramonskie]

realted to cloudfoundry/bosh-virtualbox-cpi-release#30

[ZPascal]

@chromium (Linux stack) and I (Mac OS X stack) currently working on that topic and we're documenting our efforts that inside this draft PR.

[chombium]

@ramonskie, @rkoster I think we can close this as the problem is not in this repo but in bosh-virtualbox-cpi-release and it is a dupplicate of #30 in the bosh-virtualbox-cpi-release.

[ramonskie]

closing the issue see above

[chombium]

thanks @ramonskie