Delay in pod deletion when having localpv provisioned

[lordofire]

Describe the bug: When a pod is created with the localpv provisioned, deleting the pod using kubectl along with the pv will take around 40+ seconds whereas creating a pod should return instantly.

Expected behaviour: Delete a pod should have very small blocking wait compared to pod creation.

Steps to reproduce the bug:

1. Setup the hostpath with xfs quota enabled. Install the openebs localpv provisioner.
2. Create a pod along with the pv with the following configuration test.yaml

apiVersion: apps/v1
kind: List
items:
- kind: PersistentVolumeClaim
  apiVersion: v1
  metadata:
    name: local-hostpath-pvc
    namespace: infra-offline-dev
  spec:
    storageClassName: openebs-hostpath
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 5G
- apiVersion: v1
  kind: Pod
  metadata:
    name: hello-local-device-pod
    namespace: infra-offline-dev
  spec:
    volumes:
      - name: local-storage
        persistentVolumeClaim:
          claimName: local-hostpath-pvc
    containers:
      - name: hello-container
        image: busybox
        command:
          - sh
          - -c
          - 'while true; do echo "`date` [`hostname`] Hello from OpenEBS Local PV." >> /mnt/store/greet.txt; sleep $(($RANDOM % 5 + 300)); done'
        volumeMounts:
          - mountPath: /mnt/store
            name: local-storage

cmd: kubectl apply -f test.yaml
3. After waiting the pod being created, delete the pod as well as the pvc together using kubectl:
cmd: kubectl delete -f test.yaml

My test output when performing the delete:

Linux$ time kubectl delete -f test.yaml
persistentvolumeclaim "local-hostpath-pvc" deleted
pod "hello-local-device-pod" deleted

real	0m43.641s
user	0m0.098s
sys	0m0.034s

Anything else we need to know?:
N/A

Environment details:

• OpenEBS version (use kubectl get po -n openebs --show-labels):

NAME                                            READY   STATUS    RESTARTS   AGE   LABELS
openebs-localpv-provisioner-5d88cb474b-j59qk    1/1     Running   16         32d   name=openebs-localpv-provisioner,openebs.io/component-name=openebs-localpv-provisioner,openebs.io/version=3.3.0,pod-template-hash=5d88cb474b
openebs-ndm-cluster-exporter-84bb5fc764-jxtkh   1/1     Running   0          32d   name=openebs-ndm-cluster-exporter,openebs.io/component-name=ndm-cluster-exporter,openebs.io/version=3.3.0,pod-template-hash=84bb5fc764
openebs-ndm-g2rsl                               1/1     Running   0          31d   controller-revision-hash=56bdc87c48,name=openebs-ndm,openebs.io/component-name=ndm,openebs.io/version=3.3.0,pod-template-generation=3
openebs-ndm-node-exporter-6wt6d                 1/1     Running   0          31d   controller-revision-hash=5fc9bcf946,name=openebs-ndm-node-exporter,openebs.io/component-name=ndm-node-exporter,openebs.io/version=3.3.0,pod-template-generation=3
openebs-ndm-operator-7657446466-cz4w9           1/1     Running   0          32d   name=openebs-ndm-operator,openebs.io/component-name=ndm-operator,openebs.io/version=3.3.0,pod-template-hash=7657446466

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.12", GitCommit:"696a9fdd2a58340e61e0d815c5769d266fca0802", GitTreeState:"clean", BuildDate:"2022-04-13T19:07:00Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.12", GitCommit:"696a9fdd2a58340e61e0d815c5769d266fca0802", GitTreeState:"clean", BuildDate:"2022-04-13T19:01:10Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:
  Bare-metal. XFS config is the following:

UUID="1e99f408-4103-4432-b5c0-f943400c8e6a" /data         xfs     defaults,uquota,pquota 0   0
/data/k8s/pv /var/kubernetes/pv auto bind 0 0

• OS (e.g: cat /etc/os-release):

NAME="Ubuntu"
VERSION="20.04.5 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.5 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

• kernel (e.g: uname -a):

Linux test-host 5.15.0-46-generic #49~20.04.1-Ubuntu SMP Thu Aug 4 19:15:44 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

• others:
  N/A

[lordofire]

Any update on this issue?

[lordofire]

Any update on this issue?

[niladrih]

This might help in achieving quicker Pod-schedule times for the hostpath delete job. There aren't a lot of tweak-able setting that make the controller more. However, that is an enhancement that could benefit users.

I'll see the hostNetwork change going in in v4.1.0. Maybe you could see if that helps with your times.

[tiagolobocastro]

@lordofire did you capture some logs from openebs-localpv-provisioner-5d88cb474b-j59qk at the time? It seems to have restarted 16 times.
@niladrih any updates here, do you understand the root cause?