Errors when doing concurrent host assisted clones

[akalenyu]

What happened:
Host-assisted clone source pods keep erroring when doing concurrent dumb clones.
With 2 concurrent clones it seems to converge slowly to success.

What you expected to happen:
restarts are mostly edge case on host assisted clones

How to reproduce it (as minimally and precisely as possible):
Patch storage profile to do host assisted cloning

# k get storageprofile rook-ceph-block -o yaml
selecting docker as container runtime
apiVersion: cdi.kubevirt.io/v1beta1
kind: StorageProfile
metadata:
...
  name: rook-ceph-block
spec:
  claimPropertySets:
  - accessModes:
    - ReadWriteOnce
    volumeMode: Filesystem
  cloneStrategy: copy
status:
  claimPropertySets:
  - accessModes:
    - ReadWriteOnce
    volumeMode: Filesystem
  cloneStrategy: copy
  provisioner: rook-ceph.rbd.csi.ceph.com
  storageClass: rook-ceph-block

source dv yaml

apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
  name: golden-snap-source
  namespace: golden-ns
spec:
  source:
      http:
         url: http://.../Fedora-Cloud-Base-34-1.2.x86_64.qcow2
  pvc:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 8Gi

run concurrent clones

#!/bin/bash

for (( c=0; c<2; c++ ))
do
  cat << __EOF__ | ./cluster-up/kubectl.sh create -f -
---
apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
  name: cloned-datavolume-${c}
  namespace: default
spec:
  source:
    pvc:
      namespace: golden-ns
      name: golden-snap-source
  storage:
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 9Gi
__EOF__
done

Additional context:
Errors:

I0122 14:58:03.462143      10 prometheus.go:72] 25.53
F0122 14:58:03.536422      10 clone-source.go:41] Subprocess did not execute successfully, result is: '\x02'
disk.img
/usr/bin/tar: disk.img: Read error at byte 1119744, while reading 512 bytes: Permission denied
/usr/bin/tar: Exiting with failure status due to previous errors

NAMESPACE   NAME                  PHASE             PROGRESS   RESTARTS   AGE
default     cloned-datavolume-1   CloneInProgress   24.66%     3          2m13s

Environment:

• CDI version (use kubectl get deployments cdi-deployment -o yaml): N/A
• Kubernetes version (use kubectl version): N/A
• DV specification: N/A
• Cloud provider or hardware configuration: N/A
• OS (e.g. from /etc/os-release): N/A
• Kernel (e.g. uname -a): N/A
• Install tools: N/A
• Others: N/A

[akalenyu]

Couple of notes from grooming meeting:

• This reproduces on single node cluster (the default kubevirtci install)
• We might want (regardless of this issue) set affinity on clone source pods due to Enforce ReadWriteOnce PVC access mode during scheduling kubernetes/kubernetes#103305

[kubevirt-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[akalenyu]

/remove-lifecycle stale

[alromeros]

Haven't tested this yet, but maybe it's been fixed with the recent populator refactoring?

[kubevirt-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[akalenyu]

/remove-lifecycle stale

[kubevirt-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[kubevirt-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle rotten

[kubevirt-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

[kubevirt-bot]

@kubevirt-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.