Issues while moving instances with snapshot from one storage to another

[afriqs]

Required information

• Distribution: Debian
• Distribution version: 12 (bookworm)
• The output of "incus info" or if that fails:
  • Kernel version: 6.1.0-25-amd64
  • Incus version: 6.5
  • Storage backend in use: btrfs and zfs (see below)

Issue description

While installing lxd some years ago I used a btrfs backend for storage. I wanted to try zfs so I created one and wanted to move my instances to the latest one, but my instances did not want to start properly.

storages details

$ incus storage show lxc-storage | head -6
config:
  size: 150GiB
  source: /var/lib/incus/disks/lxc-storage.img
description: ""
name: lxc-storage
driver: btrfs
$ incus storage show incus-zfs | head -7
config:
  size: 60GiB
  source: /var/lib/incus/disks/incus-zfs.img
  zfs.pool_name: incus-zfs
description: ""
name: incus-zfs
driver: zfs

Steps to reproduce

without snapshots

Everything seems to be 👍

$ incus launch images:debian/12 my-debian --vm
Launching my-debian
$ incus exec my-debian hostname
my-debian
$ incus stop my-debian
$ incus move my-debian --storage incus-zfs
$ incus start my-debian
$ incus exec my-debian hostname
my-debian

with snapshots

Something went bad

$ incus launch images:debian/12 my-debian --vm
Launching my-debian
$ incus exec my-debian hostname
my-debian
$ incus stop my-debian
$ incus snapshot create my-debian clean-install
$ incus snapshot list my-debian
+---------------+-----------------------+------------+----------+
|      NOM      |       TAKEN AT        | EXPIRES AT | STATEFUL |
+---------------+-----------------------+------------+----------+
| clean-install | 2024/09/29 19:08 CEST |            | NO       |
+---------------+-----------------------+------------+----------+
$ incus move my-debian --storage incus-zfs
$ incus start my-debian
$ incus exec my-debian hostname
Error: VM agent isn't currently running
$ incus console my-debian --type=vga
$ incus stop my-debian --force
$ incus move my-debian --storage lxc-storage
$ incus start my-debian
$ incus exec my-debian hostname
Error: VM agent isn't currently running
$ incus console my-debian --type=vga
$ incus stop my-debian --force

Information to attach

The cropped snapshots of th console :

[alex14641]

I followed all the steps OP specified, and got the same result; however, when I stopped then started the VM, waited 20 seconds, then ran the exec command, it worked.

My environment:
Distro: Slackware current
Incus version: build from latest git
Kernel version: 6.10.13
ZFS version: 2.26

[afriqs]

Hi,

I tried again with version 6.6 and it seems to work now 🤷

I moved a windows 11 vm and it fails to start with a UEFI error but this is another story I think, so I close the issue.

[afriqs]

Well, I think I was too optimistic 😅 I moved a bullseye vm I work on and it fails to boot, see screenshot

[afriqs]

Another test performed this morning:

1. backup vm with snapshots: incus export my-vm.tgz
2. remove vm: incus delete my-vm
3. import back vm: incus import my-vm.tgz (in defautl storage wich is still btrfs)
4. start vm: 👌
5. stop vm
6. remove vm: incus delete my-vm
7. import vm in zfs storage: incus import my-vm.tgz --storage incus-zfs
8. start vm: 💥

[stgraber]

Okay so what I'm reading is that the reproducer should be:

• Create a VM on a btrfs pool
• Create some snapshots
• Export the VM
• Delete the VM

Then a re-import onto btrfs works fine but a re-import on zfs doesn't?

I'm currently testing this scenario so let's see if I run into issues.

[stgraber]

Reproduced the issue here, going to have to figure out exactly what's going on here but need to sort out a much smaller test VM so things don't take quite so long to reproduce.