Failed to configure pod interface: timed out waiting for OVN port binding

[gabrilabs75]

Describe the bug

Pods that are restarted or created in a specific host/node in OKD Platform cluster are stuck in a creation loop with error

Version

Client Version: 4.9.0-0.okd-2021-11-28-035710
Server Version: 4.9.0-0.okd-2022-02-12-140851
Kubernetes Version: v1.22.1-1839+b93fd35dd03051-dirty

How reproducible
This issue happens 3 times in the last 2 months and it's really critical because we are proceeding to production release

Log bundle

28m Warning FailedCreatePodSandBox pod/apiserver-vfkpj (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_apiserver-vfkpj_openshift-apiserver **error adding pod openshift-apiserver_apiserver-vfkpj to CNI network** "multus-cni-network": [openshift-apiserver/apiserver-vfkpj/4581fd06-fc1d-4baa-9390-825666f47f74:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[openshift-apiserver/apiserver-vfkpj ] [openshift-apiserver/apiserver-vfkpj ] failed to configure pod interface: timed out waiting for OVS port binding (ovn-installed) for... 

Do you have any idea if the bug will be overcome by upgrading OKD to 4.10?

[vrutkovs]

Please attach (or upload to the public file sharing service) must-gather archive

[gabrilabs75]

Hi,
we would prefer to avoid sharing sensitive customer cluster data.
can we do it any other way?

[vrutkovs]

Remove the sensitive data and upload it? OKD is supported by community so we can't assign a particular engineer to this