Virtual machines created with cockpit: backup size not thin provisioned

[JSkolnik]

First of all I want to say thank you for this great software.
Now I want to ask. I'm testing this product and I'm running into a problem or if it's a feature.

I am trying remote backup which works. Both full backup and incremental. But the first full backup does not have thin allocation, but thick (raw).
I am backing up a VM that has a 20 GB disk, but the disk size is 2.6 GB. But the size of the full backup is a full 20 GB. Should full backup size = disk size or occupied capacity?

[abbbi]

hi,

what disk format is the virtual machine using? raw or qcow images? Thin provisioned backup and incremental backup only
works if the virtual machine uses qcow disk image format, see:

https://github.com/abbbi/virtnbdbackup#supported-disk-formats--raw-disks

[JSkolnik]

The backed up VM disk is in qcow2 format.
[root@kvm04 images]# qemu-img info rocky8.qcow2
image: rocky8.qcow2
file format: qcow2
virtual size: 20 GiB (21474836480 bytes)
disk size: 2.64 GiB
cluster_size: 65536
Format specific information:
compat: 1.1
compression type: zlib
lazy refcounts: true
refcount bits: 16
corrupt: false
extended l2: false
Child node '/file':
filename: rocky8.qcow2
protocol type: file
file length: 20.9 GiB (22398108160 bytes)
disk size: 2.64 GiB
Format specific information:
extent size hint: 1048576

Backup log:
[root@backup ~]# virtnbdbackup -U qemu+ssh://root@192.168.18.120/system --ssh-user root -d rocky8 -l auto -o /mnt/data/new
[2023-08-19 19:10:30] INFO lib common - printVersion [MainThread]: Version: 1.9.37 Arguments: /usr/bin/virtnbdbackup -U qemu+ssh://root@192.168.18.120/system --ssh-user root -d rocky8 -l auto -o /mnt/data/new
[2023-08-19 19:10:30] INFO root virtnbdbackup - main [MainThread]: Backup level: [auto]
[2023-08-19 19:10:30] INFO root virtnbdbackup - main [MainThread]: Backup mode auto, target folder is empty: executing full backup.
[2023-08-19 19:10:30] INFO virt client - _connect [MainThread]: Connected to remote host: [kvm04], local host: [backup]
[2023-08-19 19:10:30] INFO root virtnbdbackup - main [MainThread]: Libvirt library version: [9000000]
[2023-08-19 19:10:30] INFO root disktype - Optical [MainThread]: Skipping attached [cdrom] device: [sda].
[2023-08-19 19:10:30] INFO root virtnbdbackup - main [MainThread]: Backup will save [1] attached disks.
[2023-08-19 19:10:30] INFO root virtnbdbackup - main [MainThread]: Concurrent backup processes: [1]
[2023-08-19 19:10:30] INFO root checkpoint - redefine [MainThread]: Loading checkpoint list from: [/mnt/data/new/checkpoints]
[2023-08-19 19:10:30] INFO root checkpoint - create [MainThread]: Checkpoint handling.
[2023-08-19 19:10:30] INFO root checkpoint - create [MainThread]: Using checkpoint name: [virtnbdbackup.0].
[2023-08-19 19:10:30] INFO ssh client - connect [MainThread]: Connecting remote system [kvm04] via ssh, username: [root]
[2023-08-19 19:10:30] INFO paramiko.transport transport - _log [Thread-1]: Connected (version 2.0, client OpenSSH_8.7)
[2023-08-19 19:10:30] INFO paramiko.transport transport - _log [Thread-1]: Authentication (publickey) successful!
[2023-08-19 19:10:30] INFO root virtnbdbackup - main [MainThread]: Remote NBD Endpoint host: [kvm04]
[2023-08-19 19:10:30] INFO root virtnbdbackup - main [MainThread]: Temporary scratch file target directory: [/var/tmp]
[2023-08-19 19:10:30] INFO root virtnbdbackup - startBackupJob [MainThread]: Starting backup job.
[2023-08-19 19:10:30] INFO fs fs - freeze [MainThread]: Freezed [2] filesystems.
[2023-08-19 19:10:30] INFO fs fs - thaw [MainThread]: Thawed [2] filesystems.
[2023-08-19 19:10:30] INFO root virtnbdbackup - main [MainThread]: Started backup job with checkpoint, saving information.
[2023-08-19 19:10:30] INFO root checkpoint - backup [MainThread]: Saving checkpoint config to: [/mnt/data/new/checkpoints/virtnbdbackup.0.xml]
[2023-08-19 19:10:30] INFO nbd client - printVersion [vda]: libnbd version: 1.6.0
[2023-08-19 19:10:30] INFO nbd client - connect [vda]: Waiting until NBD server at [nbd://kvm04:10809/vda] is up.
[2023-08-19 19:10:31] INFO nbd client - _getBlockInfo [vda]: Using Maximum Block size supported by NBD server: [33554432]
[2023-08-19 19:10:31] INFO nbd client - connect [vda]: Connection to NBD backend succeeded.
[2023-08-19 19:10:31] INFO root virtnbdbackup - backupDisk [vda]: Got 40 extents to backup.
[2023-08-19 19:10:31] INFO root virtnbdbackup - backupDisk [vda]: 21474836480 bytes disk size
[2023-08-19 19:10:31] INFO root virtnbdbackup - backupDisk [vda]: 21468348416 bytes of data extents to backup
[2023-08-19 19:10:31] INFO root virtnbdbackup - getWriter [vda]: Write data to target file: [/mnt/data/new/vda.full.data.partial].
[2023-08-19 19:10:31] INFO root virtnbdbackup - backupDisk [vda]: Creating thin provisioned stream backup image
[2023-08-19 19:10:53] INFO root virtnbdbackup - main [MainThread]: Backup jobs finished, stopping backup task.
[2023-08-19 19:10:53] INFO root metadata - backupConfig [MainThread]: Saving VM config to: [/mnt/data/new/vmconfig.virtnbdbackup.0.xml]
[2023-08-19 19:10:53] INFO root metadata - backupDiskInfo [MainThread]: Saved qcow image config to: [/mnt/data/new/vda.virtnbdbackup.0.qcow.json]
[2023-08-19 19:10:53] INFO paramiko.transport.sftp sftp - _log [MainThread]: [chan 1] Opened sftp connection (server version 3)
[2023-08-19 19:10:53] INFO paramiko.transport.sftp sftp - _log [MainThread]: [chan 2] Opened sftp connection (server version 3)
[2023-08-19 19:10:53] INFO paramiko.transport.sftp sftp - _log [MainThread]: [chan 2] sftp session closed.
[2023-08-19 19:10:53] INFO root virtnbdbackup - main [MainThread]: Finished successfully

Backup size:
[root@backup new]# du -hs vda.full.data
20G vda.full.data

OS:
KVM server: RHEL 9
backup server: Rocky linux 8

[abbbi]

hm,, that looks all ok for me. Is the output of the qemu-img command from before or after the full backup?
Because usually this would show an bitmap that is existant, example:

file format: qcow2
virtual size: 50 MiB (52428800 bytes)
disk size: 6.13 MiB
cluster_size: 65536
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    bitmaps:
        [0]:
            flags:
                [0]: auto
            name: virtnbdbackup.0
            granularity: 65536

i wonder why the bitmap is missing in your output.

[2023-08-19 19:10:31] INFO nbd client - _getBlockInfo [vda]: Using Maximum Block size supported by NBD server: [33554432]
[2023-08-19 19:10:31] INFO nbd client - connect [vda]: Connection to NBD backend succeeded.
[2023-08-19 19:10:31] INFO root virtnbdbackup - backupDisk [vda]: Got 40 extents to backup.
[2023-08-19 19:10:31] INFO root virtnbdbackup - backupDisk [vda]: 21474836480 bytes disk size
[2023-08-19 19:10:31] INFO root virtnbdbackup - backupDisk [vda]: 21468348416 bytes of data extents to backup

virtnbdbackup uses the NBD backend to query which blocks are "dirty" and need to be backed up.
Apparently the NBD Server seems to tell that it needs to backup all 40 extents resulting in 21468348416 bytes of data.

Note there is a slight difference between the image size and the real data that is backed up, so it appears
to skip at least SOME blocks in the image:

[2023-08-19 19:10:31] INFO root virtnbdbackup - backupDisk [vda]: 21474836480 bytes disk size
[2023-08-19 19:10:31] INFO root virtnbdbackup - backupDisk [vda]: 21468348416 bytes of data extents to backup

I cant reproduce this on my systems with my qcow images. Maybe its something specific
to the qcow image you have there. It would be interesting how the image was created, or maybe the installation
process in the image did once zero all leftover blocks?

An extent to be backed up can either be "data" (real data) or "zeroes": zeroes can be skipped.
Maybe the qcow image was truncated once to the full size, so it appears to be sparse, but for the NBD Service
these zeroes are real "data" which results in the full backup size. Some cloud-init related deployments do
things like "dd if=/dev/zero of=foobar" within the image until it is "full" and then remove the file afterwards
to gain some "pre-allocation" performance features.

Can you attach backup log with verbose mode? It includes detailed informations which blockmap is returned by
the NBD service.

[abbbi]

another interesting thing would be if you backup using option:

-q, --qemu Use Qemu tools to query extents

virtnbdbackup then uses the "nbdinfo" utility from libnbd to query which data blocks need to be backed up.

If the backup has the same size using this option, then the custom implementation virtnbdbackup uses is
not at fault and the blocks in the image are really data blocks (probably zeroed out once for pre-allocation).

[JSkolnik]

Running a backup with the -q parameter has the same result.

[2023-08-20 17:18:45] INFO lib common - printVersion [MainThread]: Version: 1.9.37 Arguments: /usr/bin/virtnbdbackup -U qemu+ssh://root@192.168.18.120/system -q --ssh-user root -d rocky8 -l auto -o /mnt/data/
[2023-08-20 17:18:45] INFO root virtnbdbackup - main [MainThread]: Backup level: [auto]
[2023-08-20 17:18:45] INFO root virtnbdbackup - main [MainThread]: Backup mode auto, target folder is empty: executing full backup.
[2023-08-20 17:18:45] INFO virt client - _connect [MainThread]: Connected to remote host: [kvm04], local host: [backup]
[2023-08-20 17:18:45] INFO root virtnbdbackup - main [MainThread]: Libvirt library version: [9000000]
[2023-08-20 17:18:45] INFO root disktype - Optical [MainThread]: Skipping attached [cdrom] device: [sda].
[2023-08-20 17:18:45] INFO root virtnbdbackup - main [MainThread]: Backup will save [1] attached disks.
[2023-08-20 17:18:45] INFO root virtnbdbackup - main [MainThread]: Concurrent backup processes: [1]
[2023-08-20 17:18:45] INFO root checkpoint - redefine [MainThread]: Loading checkpoint list from: [/mnt/data//checkpoints]
[2023-08-20 17:18:45] INFO root checkpoint - create [MainThread]: Checkpoint handling.
[2023-08-20 17:18:45] INFO root checkpoint - create [MainThread]: Using checkpoint name: [virtnbdbackup.0].
[2023-08-20 17:18:45] INFO ssh client - connect [MainThread]: Connecting remote system [kvm04] via ssh, username: [root]
[2023-08-20 17:18:45] INFO paramiko.transport transport - _log [Thread-1]: Connected (version 2.0, client OpenSSH_8.7)
[2023-08-20 17:18:45] INFO paramiko.transport transport - _log [Thread-1]: Authentication (publickey) successful!
[2023-08-20 17:18:45] INFO root virtnbdbackup - main [MainThread]: Remote NBD Endpoint host: [kvm04]
[2023-08-20 17:18:45] INFO root virtnbdbackup - main [MainThread]: Temporary scratch file target directory: [/var/tmp]
[2023-08-20 17:18:45] INFO root virtnbdbackup - startBackupJob [MainThread]: Starting backup job.
[2023-08-20 17:18:46] INFO fs fs - freeze [MainThread]: Freezed [2] filesystems.
[2023-08-20 17:18:46] INFO fs fs - thaw [MainThread]: Thawed [2] filesystems.
[2023-08-20 17:18:46] INFO root virtnbdbackup - main [MainThread]: Started backup job with checkpoint, saving information.
[2023-08-20 17:18:46] INFO root checkpoint - backup [MainThread]: Saving checkpoint config to: [/mnt/data//checkpoints/virtnbdbackup.0.xml]
[2023-08-20 17:18:46] INFO nbd client - printVersion [vda]: libnbd version: 1.6.0
[2023-08-20 17:18:46] INFO nbd client - connect [vda]: Waiting until NBD server at [nbd://kvm04:10809/vda] is up.
[2023-08-20 17:18:47] INFO nbd client - _getBlockInfo [vda]: Using Maximum Block size supported by NBD server: [33554432]
[2023-08-20 17:18:47] INFO nbd client - connect [vda]: Connection to NBD backend succeeded.
[2023-08-20 17:18:47] INFO root virtnbdbackup - getExtentHandler [vda]: Using qemu tools to query extents
[2023-08-20 17:18:47] INFO root virtnbdbackup - backupDisk [vda]: Got 39 extents to backup.
[2023-08-20 17:18:47] INFO root virtnbdbackup - backupDisk [vda]: 21474836480 bytes disk size
[2023-08-20 17:18:47] INFO root virtnbdbackup - backupDisk [vda]: 21468348416 bytes of data extents to backup
[2023-08-20 17:18:47] INFO root virtnbdbackup - getWriter [vda]: Write data to target file: [/mnt/data//vda.full.data.partial].
[2023-08-20 17:18:47] INFO root virtnbdbackup - backupDisk [vda]: Creating thin provisioned stream backup image
[2023-08-20 17:19:13] INFO root virtnbdbackup - main [MainThread]: Backup jobs finished, stopping backup task.                                                                                                                                                           
[2023-08-20 17:19:13] INFO root metadata - backupConfig [MainThread]: Saving VM config to: [/mnt/data//vmconfig.virtnbdbackup.0.xml]
[2023-08-20 17:19:13] INFO root metadata - backupDiskInfo [MainThread]: Saved qcow image config to: [/mnt/data//vda.virtnbdbackup.0.qcow.json]
[2023-08-20 17:19:13] INFO paramiko.transport.sftp sftp - _log [MainThread]: [chan 1] Opened sftp connection (server version 3)
[2023-08-20 17:19:13] INFO paramiko.transport.sftp sftp - _log [MainThread]: [chan 2] Opened sftp connection (server version 3)
[2023-08-20 17:19:13] INFO paramiko.transport.sftp sftp - _log [MainThread]: [chan 2] sftp session closed.
[2023-08-20 17:19:13] INFO root virtnbdbackup - main [MainThread]: Finished successfully

[root@backup data]# du -hs vda.full.data 
20G	vda.full.data

[JSkolnik]

qemu-img info after backup looks like this:

image: rocky8.qcow2
file format: qcow2
virtual size: 20 GiB (21474836480 bytes)
disk size: 2.64 GiB
cluster_size: 65536
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: true
    bitmaps:
        [0]:
            flags:
                [0]: auto
            name: virtnbdbackup.0
            granularity: 65536
    refcount bits: 16
    corrupt: false
    extended l2: false
Child node '/file':
    filename: rocky8.qcow2
    protocol type: file
    file length: 20.9 GiB (22398108160 bytes)
    disk size: 2.64 GiB
    Format specific information:
        extent size hint: 1048576

The disk image is created with thin allocation and has not been formatted - filled with data - zeros. I created the VM from Cockpit.
Now I tried creating a new VM and the behavior is the same.

I am attaching the verbose log as an attachment.

backup.auto.08202023163941.log

[abbbi]

The disk image is created with thin allocation and has not been formatted - filled with data - zeros. I created the VM from Cockpit. Now I tried creating a new VM and the behavior is the same.

the backup uses the same extents with the qemu utils. What happens if you resize the image from 20 to 50 GB with
qemu-img, does the backup then create a 50GB backup file? I guess not. What options does cockpit use to create the
qcow image files? Im not familiar with it and havent used cockpit to create virtual machines.

I really think this has to do with the way cockpit creates the images, havent seen this behavior either on my systems
or any other users has experienced it. Are the images cloned from some image service? Can you try to create a
qemu file using qemu-img with lets say 40 GB, put a filesystem on it and see what the backup then creates?

I am attaching the verbose log as an attachment.

there is only one extend returned with data: false, which means this extend, from the image block layer does
not contain real data but is sparse and can be skipped:

 grep data.*false backup.auto.08202023163941.log
        "data": false,

all other extents returned from the NBD service contain data: true, which means its actual data from the qemu image
block layers point of view. I really guess these images have been zeroed out once (dd if=/dev/zero of=tmpfile). This results in the
situation that the zeroes written to the image appear as sparse file to the file system, but from the qemu block layer
these extents have been written by the operating system, thus are these data blocks have been changed and need
to be backed up as such. What file system is used within the virtual machine?

Alternative may be that there is an bug in qemu/libvirt in the versions your are using, from virtnbdbackups point of view
, everything works like expected, the data were told to back up is saved. Example from the virtual machine used as CI test:

qemu-img info /tmp/tmp.wTMLVQpC7y/vm1-sda.qcow2  | grep size
virtual size: 50 MiB (52428800 bytes)
disk size: 6.13 MiB
./virtnbdbackup -d vm1 -l full -o /tmp/SIZE
 ls -alh /tmp/SIZE/sda.full.data 
-rw-r--r-- 1 abi abi 5.9M Aug 20 17:35 /tmp/SIZE/sda.full.data

[abbbi]

Maybe cockpit uses some other tools (virt-*) in the process to create so called "thin eager provisioned" image files where the zeroes are pre-allocated but sparse, ..

Edit: virtual machine configuration as from "virsh dumpxml " would help too.

[JSkolnik]

the backup uses the same extents with the qemu utils. What happens if you resize the image from 20 to 50 GB with
qemu-img, does the backup then create a 50GB backup file?

I did a resize +10G > then a new full backup. The size of the backup is 21GB (not 30GB like the virtual disk).

I created a new disk via cli - qemu-img ... > new full backup. The backup size is 2.1GB. Image size is 10GB. In this case, thin provisioning is OK.

This is how it looks like Cockpit really does something different.

Edit: virtual machine configuration as from "virsh dumpxml " would help too.

<domain type='kvm' id='22'>
  <name>rocky8</name>
  <uuid>d47beb29-baa7-4ad1-b851-77900f971456</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://rockylinux.org/rocky/8"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit='KiB'>1572864</memory>
  <currentMemory unit='KiB'>1572864</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-q35-rhel9.2.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'/>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/libvirt/images/rocky8.qcow2' index='3'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </disk>
    <disk type='volume' device='disk'>
      <driver name='qemu' type='qcow2' cache='none' discard='unmap'/>
      <source pool='ssd_sas' volume='rocky08-2.qcow2' index='2'/>
      <backingStore/>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu'/>
      <target dev='sda' bus='sata'/>
      <readonly/>
      <alias name='sata0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x14'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0x15'/>
      <alias name='pci.6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0x16'/>
      <alias name='pci.7'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/>
    </controller>
    <controller type='pci' index='8' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='8' port='0x17'/>
      <alias name='pci.8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x7'/>
    </controller>
    <controller type='pci' index='9' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='9' port='0x18'/>
      <alias name='pci.9'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='10' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='10' port='0x19'/>
      <alias name='pci.10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1'/>
    </controller>
    <controller type='pci' index='11' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='11' port='0x1a'/>
      <alias name='pci.11'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x2'/>
    </controller>
    <controller type='pci' index='12' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='12' port='0x1b'/>
      <alias name='pci.12'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x3'/>
    </controller>
    <controller type='pci' index='13' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='13' port='0x1c'/>
      <alias name='pci.13'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x4'/>
    </controller>
    <controller type='pci' index='14' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='14' port='0x1d'/>
      <alias name='pci.14'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x5'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:ab:1d:6f'/>
      <source bridge='bridge0'/>
      <target dev='vnet22'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/0'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/0'>
      <source path='/dev/pts/0'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-22-rocky8/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'>
      <alias name='input1'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input2'/>
    </input>
    <graphics type='vnc' port='5900' autoport='yes' listen='127.0.0.1'>
      <listen type='address' address='127.0.0.1'/>
    </graphics>
    <audio id='1' type='none'/>
    <video>
      <model type='virtio' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </memballoon>
    <rng model='virtio'>
      <backend model='random'>/dev/urandom</backend>
      <alias name='rng0'/>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </rng>
  </devices>
  <seclabel type='dynamic' model='selinux' relabel='yes'>
    <label>system_u:system_r:svirt_t:s0:c467,c870</label>
    <imagelabel>system_u:object_r:svirt_image_t:s0:c467,c870</imagelabel>
  </seclabel>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+107:+107</label>
    <imagelabel>+107:+107</imagelabel>
  </seclabel>
</domain>

Disk vda create via Cockpit
Disk vdb create via cli

[abbbi]

I created a new disk via cli - qemu-img ... > new full backup. The backup size is 2.1GB. Image size is 10GB. In this case, thin provisioning is OK.

This is how it looks like Cockpit really does something different.

yes, might depend on the installtion method used. Cockpit seems to use virt-install under the hood.
If the "Download OS" option is used a prepared image is downloaded which is then adjusted in it size for the virtual disk chosen.
Not sure how they re-size or cloning the images but that seems to result in the situation youre having.