OKD 4.9 - Pods on on-premise worker node are unable to communicate with the ones on remote worker node

[imdmahajankanika]

Describe the bug

In our OKD 4.9 cluster with one on-premise worker (worker-0) and one remote worker node (rm-worker-0), the pods running on worker-0 are unable to connect with the ones running on rm-worker-0 inside the cluster (through service) and vice-versa. However, we have tested that the communication is possible if we add "hostNetwork=true" in the deployment.

Version
4.9.0-0.okd-2022-02-12-140851

How reproducible

1. We deployed our application (php) on pod using (worker-0) and the database (mysql) on remote worker (rm-worker-0). But the application is unable to access database.
2. Also, if an application pod is deployed on remote-worker, it is unable to pull image from internal registry.

Log bundle

We tried to do curl to internal image registry from remote-worker, but it is giving below error.

• Trying 172.30.84.37:5000...
• connect to 172.30.84.37 port 5000 failed: No route to host
• Failed to connect to image-registry.openshift-image-registry.svc port 5000: No route to host
• Closing connection 0
  curl: (7) Failed to connect to image-registry.openshift-image-registry.svc port 5000: No route to host