Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Submariner 0.12.0 not able to establish connection between sites. Error: The detected CNI network plugin ("") is not supported by Submariner. #3042

Open
BhavaniYalamanchili opened this issue Jun 7, 2024 · 22 comments
Assignees

Comments

@BhavaniYalamanchili
Copy link

BhavaniYalamanchili commented Jun 7, 2024

ISSUE:

The Submariner is not able to establish the connection between the sites.
The error we are seeing in the diagnose all command is
Error Message: The detected CNI network plugin ("") is not supported by Submariner. Supported network plugins: [generic canal-flannel weave-net OpenShiftSDN OVNKubernetes calico]

SETUP:

Site 1: OCP 4.10
Site 2: OCP 4.10
Submariner version: v0.12.0

We are using OVNKubernetes
From the output of oc get network/cluster -o yaml
We can see network type as this
networkType: OVNKubernetes

Outputs of show all and diagnose all commands

Show all for Site 1

Cluster "site1"
 ✓ Detecting broker(s)
NAMESPACE                NAME                     COMPONENTS
submariner-k8s-broker    submariner-broker        service-discovery, connectivity

 ✓ Showing Connections
GATEWAY                          CLUSTER  REMOTE IP       NAT  CABLE DRIVER  SUBNETS                       STATUS  RTT avg.
control-1-ru2.ocp-psb-01.gbbper  site2    192.168.126.30  no   libreswan     172.31.0.0/16, 10.132.0.0/14  error   0s

 ✓ Showing Endpoints
CLUSTER ID                    ENDPOINT IP     PUBLIC IP       CABLE DRIVER        TYPE
site1                         192.168.54.30   192.168.54.30   libreswan           local
site2                         192.168.126.30  192.168.126.30  libreswan           remote
site1                         192.168.54.31   192.168.54.31   libreswan           local
site1                         192.168.54.32   192.168.54.32   libreswan           local

 ✓ Showing Gateways
NODE                            HA STATUS       SUMMARY
control-1-ru2.ocp-psa-01.gbbper active          0 connections out of 1 are established
control-1-ru3.ocp-psa-01.gbbper passive         There are no connections
control-1-ru4.ocp-psa-01.gbbper passive         There are no connections

    Discovered network details via Submariner:
 ✓ Showing Network details
        Network plugin:
        Service CIDRs:   []
        Cluster CIDRs:   []

 ✓ Showing versions
COMPONENT                       REPOSITORY                                            VERSION
submariner                      quay.io/submariner                                    0.12.0
submariner-operator             quay.io/submariner                                    0.12.0
COMPONENT                       REPOSITORY                                            VERSION
submariner                      quay.io/submariner                                    0.12.0
submariner-operator             quay.io/submariner                                    0.12.0

Show all for Site 2

Cluster "local-config"
 ✓ Detecting broker(s)

 ✓ Showing Connections
GATEWAY                          CLUSTER  REMOTE IP      NAT  CABLE DRIVER  SUBNETS                       STATUS     RTT avg.
control-1-ru2.ocp-psa-01.gbbper  site1    192.168.54.30  no   libreswan     172.30.0.0/16, 10.128.0.0/14  connected  988.791µs

 ✓ Showing Endpoints
CLUSTER ID                    ENDPOINT IP     PUBLIC IP       CABLE DRIVER        TYPE
site2                         192.168.126.30  192.168.126.30  libreswan           local
site1                         192.168.54.30   192.168.54.30   libreswan           remote
site2                         192.168.126.31  192.168.126.31  libreswan           local
site2                         192.168.126.32  192.168.126.32  libreswan           local

 ✓ Showing Gateways
NODE                            HA STATUS       SUMMARY
control-1-ru2.ocp-psb-01.gbbper active          All connections (1) are established
control-1-ru3.ocp-psb-01.gbbper passive         There are no connections
control-1-ru4.ocp-psb-01.gbbper passive         There are no connections

    Discovered network details via Submariner:
 ✓ Showing Network details
        Network plugin:  OVNKubernetes
        Service CIDRs:   [172.31.0.0/16]
        Cluster CIDRs:   [10.132.0.0/14]

 ✓ Showing versions
COMPONENT                       REPOSITORY                                            VERSION
submariner                      quay.io/submariner                                    0.12.0
submariner-operator             quay.io/submariner                                    0.12.0
service-discovery               quay.io/submariner                                    0.12.0
COMPONENT                       REPOSITORY                                            VERSION
submariner                      quay.io/submariner                                    0.12.0
submariner-operator             quay.io/submariner                                    0.12.0
service-discovery               quay.io/submariner                                    0.12.0

Diagnose all for Site 1

sh-4.4$ /root/.local/bin/subctl diagnose all --kubeconfig site-1-kubeconfig
Cluster "site1"
 ✓ Checking Submariner support for the Kubernetes version
 ✓ Kubernetes version "v1.23.12+8a6bfe4" is supported

 ✗ Checking Submariner support for the CNI network plugin
 ✗ The detected CNI network plugin ("") is not supported by Submariner. Supported network plugins: [generic canal-flannel weave-net OpenShiftSDN OVNKubernetes calico]


 ✗ Checking gateway connections
 ✗ Connection to cluster "site2" is not established

 ✗ Checking Submariner pods
 ✗ Error obtaining Daemonset "submariner-routeagent": daemonsets.apps "submariner-routeagent" not found
 ✗ Error obtaining Deployment "submariner-lighthouse-agent": deployments.apps "submariner-lighthouse-agent" not found
 ✗ Error obtaining Deployment "submariner-lighthouse-coredns": deployments.apps "submariner-lighthouse-coredns" not found

 ✓ Non-Globalnet deployment detected - checking if cluster CIDRs overlap
 ✓ Clusters do not have overlapping CIDRs

 ✓ Checking Submariner support for the kube-proxy mode
 ✓ The kube-proxy mode is supported

 ✓ Checking the firewall configuration to determine if the metrics port (8080) is allowed
 ✓ The firewall configuration allows metrics to be retrieved from Gateway nodes

 ✗ Checking the firewall configuration to determine if VXLAN traffic is allowed
 ✗ The tcpdump output from the sniffer pod does not contain the expected remote endpoint IP 172.31.0.0. Please check that your firewall configuration allows UDP/4800 traffic.

 ✓ Globalnet is not installed - skipping

Skipping inter-cluster firewall check as it requires two kubeconfigs. Please run "subctl diagnose firewall inter-cluster" command manually.

Diagnose all for Site 2

sh-4.4$ /root/.local/bin/subctl diagnose all --kubeconfig site-2-kubeconfig
Cluster "local-config"
 ✓ Checking Submariner support for the Kubernetes version
 ✓ Kubernetes version "v1.23.12+8a6bfe4" is supported

 ✓ Checking Submariner support for the CNI network plugin
 ✓ The detected CNI network plugin ("OVNKubernetes") is supported

 ✓ Checking gateway connections
 ✓ All connections are established

 ✓ Checking Submariner pods
 ✓ All Submariner pods are up and running

 ✓ Non-Globalnet deployment detected - checking if cluster CIDRs overlap
 ✓ Clusters do not have overlapping CIDRs

 ✓ Checking Submariner support for the kube-proxy mode
 ✓ The kube-proxy mode is supported

 ✓ Checking the firewall configuration to determine if the metrics port (8080) is allowed
 ✓ The firewall configuration allows metrics to be retrieved from Gateway nodes

 ✓ Checking the firewall configuration to determine if VXLAN traffic is allowed
 ✓ This check is not necessary for the OVNKubernetes CNI plugin
 ✓ The firewall configuration allows VXLAN traffic

 ✓ Globalnet is not installed - skipping

Skipping inter-cluster firewall check as it requires two kubeconfigs. Please run "subctl diagnose firewall inter-cluster" command manually.
@yboaron
Copy link
Contributor

yboaron commented Jun 9, 2024

A. You can read how Submariner detects if CNI is OVN-K8S here (in 0.12 release)

B. I can see that OVN-K8S was detected successfully for site2 , both site1 and site2 are running OCP 4.10, is there any difference in OVN-K8S configuration between these clusters ?

@yboaron yboaron self-assigned this Jun 10, 2024
@dfarrell07
Copy link
Member

FYI, SubM 0.12 is a very old version that's long since not supposed to be supported. If you get a chance to update, that would likely be a very good idea.

@BhavaniYalamanchili
Copy link
Author

@yboaron
Based on the function code, I have listed out some oc commands to verify the pods and services the function is fetching and some output related to submariner CR
Here are the outputs:

  1. oc get pods -A -1 name=ovnkube-db
No resources found

On both sites, it is the same output
2. oc get pods -n openshift-ovn-kubernetes
Site 1

NAME                   READY   STATUS    RESTARTS   AGE
ovnkube-master-42r22   6/6     Running   34         325d
ovnkube-master-fbn4v   6/6     Running   6          37d
ovnkube-master-kbs2d   6/6     Running   6          37d
ovnkube-node-7nxcq     5/5     Running   30         325d
ovnkube-node-bcshb     5/5     Running   267        325d
ovnkube-node-br4cf     5/5     Running   25         325d
ovnkube-node-bsb2b     5/5     Running   41         325d
ovnkube-node-dpt65     5/5     Running   30         324d
ovnkube-node-fb7rw     5/5     Running   57         325d
ovnkube-node-g6jsc     5/5     Running   25         325d
ovnkube-node-jkdt8     5/5     Running   30         325d
ovnkube-node-qllh7     5/5     Running   25         325d
ovnkube-node-rb2bt     5/5     Running   25         324d
ovnkube-node-tc4qq     5/5     Running   5          37d
ovnkube-node-w7rzd     5/5     Running   25         325d

Site 2

NAME                   READY   STATUS    RESTARTS       AGE
ovnkube-master-ccnvn   6/6     Running   4 (4d3h ago)   4d3h
ovnkube-master-sfjvh   6/6     Running   4 (4d3h ago)   4d3h
ovnkube-master-xqbxp   6/6     Running   0              4d3h
ovnkube-node-48rwm     5/5     Running   0              4d3h
ovnkube-node-852x6     5/5     Running   0              4d3h
ovnkube-node-88kqh     5/5     Running   0              4d3h
ovnkube-node-bn5s6     5/5     Running   0              4d3h
ovnkube-node-frd5g     5/5     Running   0              4d3h
ovnkube-node-g6fc4     5/5     Running   0              4d3h
ovnkube-node-hx8hb     5/5     Running   0              4d3h
ovnkube-node-kp6l8     5/5     Running   0              4d3h
ovnkube-node-ld7xj     5/5     Running   0              4d3h
ovnkube-node-n89hp     5/5     Running   0              4d3h
ovnkube-node-pbmpk     5/5     Running   0              4d3h
ovnkube-node-tv8sc     5/5     Running   0              4d3h
  1. oc get services -n openshift-ovn-kubernetes | grep ovnkube-db
    Site 1
ovnkube-db              ClusterIP   None         <none>        9641/TCP,9642/TCP   325d

Site 2

ovnkube-db              ClusterIP   None         <none>        9641/TCP,9642/TCP   299d
  1. oc get submariner submariner -n submariner-operator -o yaml
    Site 1 - CR in Site 1 is showing till the Spec only, I am wondering why the status is not shown.
# oc get submariner submariner -n submariner-operator -o yaml
apiVersion: submariner.io/v1alpha1
kind: Submariner
metadata:
  creationTimestamp: "2024-06-06T13:22:34Z"
  finalizers:
  - controllers.submariner.io/cleanup
  generation: 1
  managedFields:
  - apiVersion: submariner.io/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        .: {}
        f:broker: {}
        f:brokerK8sApiServer: {}
        f:brokerK8sApiServerToken: {}
        f:brokerK8sCA: {}
        f:brokerK8sRemoteNamespace: {}
        f:brokerK8sSecret: {}
        f:ceIPSecDebug: {}
        f:ceIPSecIKEPort: {}
        f:ceIPSecNATTPort: {}
        f:ceIPSecPSK: {}
        f:ceIPSecPSKSecret: {}
        f:clusterCIDR: {}
        f:clusterID: {}
        f:connectionHealthCheck:
          .: {}
          f:enabled: {}
          f:intervalSeconds: {}
          f:maxPacketLossCount: {}
        f:debug: {}
        f:namespace: {}
        f:natEnabled: {}
        f:repository: {}
        f:serviceCIDR: {}
        f:serviceDiscoveryEnabled: {}
        f:version: {}
    manager: subctl
    operation: Update
    time: "2024-06-06T13:22:34Z"
  - apiVersion: submariner.io/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .: {}
          v:"controllers.submariner.io/cleanup": {}
    manager: submariner-operator
    operation: Update
    time: "2024-06-06T13:25:22Z"
  name: submariner
  namespace: submariner-operator
  resourceVersion: "2020929472"
  uid: e6f8ad61-63bc-4d8a-8e56-2d7ad292a2d1
spec:
  broker: k8s
  brokerK8sApiServer: api.ocp-psa-01.gbbper.priv:6443
  brokerK8sApiServerToken: eyJhbGciOiJSUzI1NiIsImtpZCI6IngwR2pZY2NWTXJtRlI2T3pxVjFzSHU5OU5SNm9fUkt3V1BMTFJxcnV3LVkifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJzdWJtYXJpbmVyLWs4cy1icm9rZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoiY2x1c3Rlci1zaXRlMS10b2tlbi1ma2NqciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJjbHVzdGVyLXNpdGUxIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiZTYxOTM4ZTItNDgxMC00MGZkLTlmMTctZGQ2NDg3NTM5NThkIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OnN1Ym1hcmluZXItazhzLWJyb2tlcjpjbHVzdGVyLXNpdGUxIn0.Z_KAnNtLYHlPRjm7x5ZSwWLAoMyd3Pq_Z9z4LD3iwxrmgfExt5iltotssdeQXDkYvOHYWL7f5XTT7FcvBhKt57aQ9s5tDvVz1LIaehwnMnHwwdYadhSDBgfvrEbAmCByxx0CXOmejuBS5sov5oIhJSeSZzT1wJ6xVsEg-pyQbewb1All0oQIaEmtQtzpWg4OeL2WAxf2u-SG2Er8RbWj_EZ5fwgAHwU2RabvN59-JhA1jE9ZoK_4fkUNlmajDYHr8l_LwtDd4ScmY7hTBARBH6uTkTx3H787qsp7zHgSDxhofT1M51ABdhea3WaWGjqefb-nWOOenSl9KyAVyqF-b_dOD_6jP3pktDE2BPD0Aj2ehOd6FoGorL7ZyYbo1-oHJusC1pE1B46K8Ij7eI_6iSkds5etONhwVonjlPt_L8vdIqn9rBaVJhHbEwz1sHrHvkKd9G0Ka41SLEjZtI7TUkTWTGUF41ViAF2D4OGIYyuiJO9YIUQu-a1tTt8bLpu-_67DHYHA2jZh510aRohUgKthUxKC7E4atVlSwYZk5Lm-5r4kscRb6r2SqIxy2EOif1dtcnpiECYgnvBFtGa1MRbna_f2VRh6KXlOWrtzAZmFIe2pE6BftIidknync1ZAXfJz-zzNsQ_YFVBzsGqiQnfUn_Opr9VWI7TSSgIgYgo
  brokerK8sCA: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURNakNDQWhxZ0F3SUJBZ0lJZkRtandjUGF1K1F3RFFZSktvWklodmNOQVFFTEJRQXdOekVTTUJBR0ExVUUKQ3hNSmIzQmxibk5vYVdaME1TRXdId1lEVlFRREV4aHJkV0psTFdGd2FYTmxjblpsY2kxc1lpMXphV2R1WlhJdwpIaGNOTWpNd056SXdNVFV3TVRFMFdoY05Nek13TnpFM01UVXdNVEUwV2pBM01SSXdFQVlEVlFRTEV3bHZjR1Z1CmMyaHBablF4SVRBZkJnTlZCQU1UR0d0MVltVXRZWEJwYzJWeWRtVnlMV3hpTFhOcFoyNWxjakNDQVNJd0RRWUoKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTGNiT0dQbnRaTE1sR2xPQ1FHODFnWCt6dG1FMUFZWgp2RUkybFM4UCtCOGVkSDJCdk9GYkc2dWc1WUhSN1h4RkdlNVllSjZ4Tkw5M01OS05uK1BLcGhrNm5kdlJDTjZtCkF0RFI3WnBBczFYVGZEOHNHOHg3TWFYZTF5RU9SdmtZQXBSLzVOSFFhSnZzUWxqaXY3blQvUWNLTFRKbXpBUUsKS0ZwQ1pLd3NVc1R0NEtsYXVPLzlDWnZpMVZlcTV0NlJhTE1xdjZyREREUC8zcHo4U0NzTlYvR3hIeGhXVWl4dQpVK2RxbmF2RmxGa0xOdys0Qnh1SnFpcUJHNGNwV2p5cjFEMmJBcThvVnoxN1JaTmw2M09VdUxGa3VYdWlsUVBmCmkzSXN1WDQ2WTE0T01icHRzTTltNzZmdElSUWJNU0paZWxqdmlqVG9VKzZ2bFRNak0vUDdkQ0VDQXdFQUFhTkMKTUVBd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3SFFZRFZSME9CQllFRkEwRwpyM2FvSk1UdVpmR3JjZlQrYm94NFZpQkdNQTBHQ1NxR1NJYjNEUUVCQ3dVQUE0SUJBUUE5dkRWRlpFOXhpUzRxCndaQS9DaFRjOG9IMDlCZmlBNTlpNjd0cldLMzZVVGo0SFZiM2ZMOEFXeXBnM2tZSGh2azZ2cUJnRHVmVFpOUjcKU2Y1OS9SY0w1VVRSdTA4bzlETHNvSTlFWUxkMjlRcTdjdHBsUmw1RmZ3UjY5ZjlOL2kwOFlRZDVKLzBLNHA1OAorR1M0WWhTR1d6c2oxUnZ2MUJBTkY2a056MFhGOG1lSVJVWkcyZFBBaFoyVWRxMnBPQ3d2MmFqZHZqWk14ZFBFCkdPMStSbDdBa3hNTEhKbXZDOGJVbEM2U3hsYm1BV2FiaTY5M3VkWjZUQkJwTjRwUkFNZnQvOHFzM3ZsRTlpMzcKajFTWitMTVNQR2xINWlwMUMzbjlDNlUxRlpBQU9vN09uYXJaSk9yZ09ua2h4MVBwM09wTDlYY1I3Q0Y2eE5sMQowbWtYcnY5aQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCi0tLS0tQkVHSU4gQ0VSVElGSUNBVEUtLS0tLQpNSUlEUURDQ0FpaWdBd0lCQWdJSUVsSU52VkpFRVBVd0RRWUpLb1pJaHZjTkFRRUxCUUF3UGpFU01CQUdBMVVFCkN4TUpiM0JsYm5Ob2FXWjBNU2d3SmdZRFZRUURFeDlyZFdKbExXRndhWE5sY25abGNpMXNiMk5oYkdodmMzUXQKYzJsbmJtVnlNQjRYRFRJek1EY3lNREUxTURFeE5Gb1hEVE16TURjeE56RTFNREV4TkZvd1BqRVNNQkFHQTFVRQpDeE1KYjNCbGJuTm9hV1owTVNnd0pnWURWUVFERXg5cmRXSmxMV0Z3YVhObGNuWmxjaTFzYjJOaGJHaHZjM1F0CmMybG5ibVZ5TUlJQklqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUF2Qy82VVNrZEhkT3MKb2NuMFBLdm52RUtEcTZ3bzRkNXJMVGUybG5jam5UVWlRQUFtNkd5YkJRVC9BNTZyb3kxSUpPNGtpWXlscFFBRQpQaEk0dkIzMGdxNjZLS0czdlpkc3dpemt6OTFEVE9wV05ZY3Bmbzh4R3ZzZFZSM1RudVNnREdCVHV3NGNCYzV3Ci9LRXlnS3BtZk1WenJiS0cxbnlNY1daUHhBTkM4V3kyRm1NTSt1eVFFaVRCak9mVVBvWTZkZVVqUXpNZkp5dGsKR1NjMWordEUwejFsWkRZMjJQMW5yTDh3TkR3Y0R3T0gydGcvcHhVV2VkeEQ0SGFLMGwzRjQvOVlxYW56T21oRQo5Q2dRaEtHTGdERzUxUWE0WU1qWUl2bTJqUExvNUpOaU05SjlkZzhKSEdxNm5SM2hkZmI4TkhjY1B4bWRtRzhjClEwWVlYUWMwcXdJREFRQUJvMEl3UURBT0JnTlZIUThCQWY4RUJBTUNBcVF3RHdZRFZSMFRBUUgvQkFVd0F3RUIKL3pBZEJnTlZIUTRFRmdRVVNBZzgvY29iR1lUaitDb0xtMGdaaW9TUnljd3dEUVlKS29aSWh2Y05BUUVMQlFBRApnZ0VCQUtDYkVIN2lKQVN1ajRVcHo0ajd4RVBzY05RS0tOS3RzQ29TbHpuWTBuRTVYUjhwV3VVdmxvTS9xdVVQClUrL3RSWldTaXV3SFdEeUJHY2tubUJhOS92aE1HVzNNcjlCOWxlL0srVHpSQlg0aGF6YWRDbXU2b0p6K1lxWTcKRkQ4elVqYkxZN3BBQ1BFdkpJY3FRWTJTRDdlODd5N0Jnam5UM3lUMzdyd01pcXV6bHI1QmNjLzg3WjRCanNnTAo4MjAyUXVFYml6Q3ZpTnBqYUNyamw4YWtKTG1SRnZZVERiaTk0bzh1YklyVHZ6a1NpbmNhWnQ5eHlibWhySEt4CmhPWjd3Tnh4VzNQRUU5RVZRN1c3RVhLYUhadk1sVWZNUXVrSS84Y05uWDltdm9wSUY5MEVrb3hkd2JQZ00yZ3gKVXN5TllpRENkWENVcGFJWUJFNTUrbWQyb3pJPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCi0tLS0tQkVHSU4gQ0VSVElGSUNBVEUtLS0tLQpNSUlEVERDQ0FqU2dBd0lCQWdJSVc4K0ZZTXhsdklJd0RRWUpLb1pJaHZjTkFRRUxCUUF3UkRFU01CQUdBMVVFCkN4TUpiM0JsYm5Ob2FXWjBNUzR3TEFZRFZRUURFeVZyZFdKbExXRndhWE5sY25abGNpMXpaWEoyYVdObExXNWwKZEhkdmNtc3RjMmxuYm1WeU1CNFhEVEl6TURjeU1ERTFNREV4TkZvWERUTXpNRGN4TnpFMU1ERXhORm93UkRFUwpNQkFHQTFVRUN4TUpiM0JsYm5Ob2FXWjBNUzR3TEFZRFZRUURFeVZyZFdKbExXRndhWE5sY25abGNpMXpaWEoyCmFXTmxMVzVsZEhkdmNtc3RjMmxuYm1WeU1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0MKQVFFQXhLTWdMcmtyc2dSaE4zZEY1TFlBZForeFdmZXNJR0laa0FVZ25tV1Y3azdVbjJCUVkvbUNuekVPTHhibgpDNm56MWFKT3R1SDNoZWg4bE5MVGIxc1JQZkFpaVBIdVBOaDRrZXMxcDdiajVHcXd2MmRNc0tESUw2aHAxbE4xClJEWFNiZUw1ajd6UjFXbUJrL01taUQ2L2wwaWpIZGRjMXpRNmlFQ2p6MXFsbHl6UDM1ZW1VS1dwcG5iQld0S2IKYjJXNGlEdVhBQ0JvbWgzVTBkalZWYitYR3JiQkhaQ21iaWMrRWcvYi8ya3J2WFpZTG93N0QwTjVaRkRkZG84TgpRcFNabTVaOUZYNHE0LzJqMjVZUjAzSGNFd1RwNVI3UXFsWDYxWUZuT2dJWUNQNzB2d0N5aXpVclE4S3dnNGZ6ClZDaGl5Q1pCSXlHYzUrR3hzcUViZXVmL2VRSURBUUFCbzBJd1FEQU9CZ05WSFE4QkFmOEVCQU1DQXFRd0R3WUQKVlIwVEFRSC9CQVV3QXdFQi96QWRCZ05WSFE0RUZnUVVGamJyS3Y0NTFiWmgvaDM1SU9hREtuUnBpd013RFFZSgpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFKK2kzVURUN3Y1cVFpS1gzSW96Qk5MVzBiN3Fncy9tUjJueGFnOWo1RFFuCkZUelRGYjl6dDVUNG5VQ0hwdVRtdjZudWFvUWhrL1BUV2RDV21uSXp5WmVjMERxcCtWWXFCQnFyZGFKUU00b0oKVFdxQTU1WFZGU0s2aVpWS09wNDNETHg2aVhMN1NEWXNiQ000d1I2a3dSbEtib3Z3NjN4SE42QWtWcCtQRzk5dQo3RkR4REU5RG00SW1zd3hhcGFEMnFZTlhkdjRpS1R5cmtHWUg2elM1VmQwSUxtSkFaQjZxMk5sUnpEb2tSY3NTCkovRUJwU3pjVVFuUlRjbkkxM21VMTFqelZRVFRMYktsVXV2d01LU3ZPa0c5bXVsU1p3SUc0RnR5SVd5d1dGbjEKQ093ZUNlaTh5K0FpaE5BYkdET3U1Q0h6WGl3dDVxYko3T3hCeGY2RUh6bz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQotLS0tLUJFR0lOIENFUlRJRklDQVRFLS0tLS0KTUlJRGx6Q0NBbitnQXdJQkFnSUlNRmpCOEN4eWczWXdEUVlKS29aSWh2Y05BUUVMQlFBd1dURlhNRlVHQTFVRQpBd3hPYjNCbGJuTm9hV1owTFd0MVltVXRZWEJwYzJWeWRtVnlMVzl3WlhKaGRHOXlYMnh2WTJGc2FHOXpkQzF5ClpXTnZkbVZ5ZVMxelpYSjJhVzVuTFhOcFoyNWxja0F4TmpnNU9EWTNPRFV6TUI0WERUSXpNRGN5TURFMU5EUXgKTWxvWERUTXpNRGN4TnpFMU5EUXhNMW93V1RGWE1GVUdBMVVFQXd4T2IzQmxibk5vYVdaMExXdDFZbVV0WVhCcApjMlZ5ZG1WeUxXOXdaWEpoZEc5eVgyeHZZMkZzYUc5emRDMXlaV052ZG1WeWVTMXpaWEoyYVc1bkxYTnBaMjVsCmNrQXhOamc1T0RZM09EVXpNSUlCSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQTNCbTUKckxjdS9kcExrczZNcS90TGxvbjJTMFM2aXNzcEZSb1FRZjNwZXg2WDNHN0NLOWpQYldsQ0dWeElpKzdqTW9XYQpHYTYwc3hmeWZ6L0NDMlZpWnJBNUMrUU1GeTJoc29kalFxeEVDNWNVVERmaHRiVzVqYWltY25xUUgrQmRXeHhECmN5c0NWQVNaWFZ4WmRvaVhqTXAwb3A2QWkydzBZekE3UzJDSWFKaTBsTjNBamNFalFRWnNPTEJiMGNrQjFrWHkKNXJMWGI3cTR0WkxQVmtzVm94dncrOUFDNzdxdkNreGwzYWd6TEowZGRjYnFXUjEvS29OdHlRMklpb0pWVmtregoyUHB2ekJ1S3hYUWR0d2s5dWwzWWhwYWIvMEhxM2VjYlpyUGZmY2VVa3RZUjc2cW9RUzhBL0tQNFI1SG5RN3VZClJGUDdQbE9hS3BqdUNqVTVuUUlEQVFBQm8yTXdZVEFPQmdOVkhROEJBZjhFQkFNQ0FxUXdEd1lEVlIwVEFRSC8KQkFVd0F3RUIvekFkQmdOVkhRNEVGZ1FVV3BIQmY2cndMRlpZVXVpYXdQU2VOSHZycDBzd0h3WURWUjBqQkJndwpGb0FVV3BIQmY2cndMRlpZVXVpYXdQU2VOSHZycDBzd0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFLS2o4NW9MClU5a1JLdDlWRVdSNVp4UmdIYzc5TDkrckhYTldxVGpRV2xUcEhrd1B5RE5wLy83OHpUdmNLbkREaEZ1dG0yTE8KMHZCbnlBNlZ3QmdWQS9hR1BnUENGUkxRZk9ndEtHNVFhTW8zUC9ZR3RyN0NHb1NpYUFPblRTZjRybjFlVy9oUAovOGlOZEFzZzVJYU5IMFkrcGM1dEVkUDhUOUNpQmUwUDViNGk3ZFNidXVBVFJOK2U0b2ZvRjV5dEd3RHJ5RjZqCm9MSi9ZUHZFSEtWanhxUWpCYkltdzVVMHJMSWpSZEQ3dlMybmNEWm9YY3Q1djBzeUxuN3JabWd0aWJJRkpFYTAKVng5c2RETkZGSWxWcHBYVXU4RnJ2RHY4TUdsNk9CQUVSZXBCQjhndVpsS0R5TlRNaU1DM3hRTjB0N2wxaXNGagprSWN2ZFFuL1dnV3I2cmc9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0KLS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUVtVENDQTRHZ0F3SUJBZ0lDQzI0d0RRWUpLb1pJaHZjTkFRRUxCUUF3Z2JjeEN6QUpCZ05WQkFZVEFrbFUKTVE4d0RRWURWUVFJRXdaTmIyUmxibUV4RnpBVkJnTlZCQWNURGtObGJuUnlieUJ6WlhKMmFYcHBNUlF3RWdZRApWUVFLRXd0SGNuVndjRzhnUWxCRlVqRVdNQlFHQTFVRUN4TU5RbEJGVWlCVFpYSjJhV05sY3pFVk1CTUdBMVVFCkRCTU1ZMkVnWW5CbGNpQXlNRE00TVRrd053WURWUVFERXpCSGNuVndjRzhnUWxCRlVpQXRJRk5sY25acGVtbHYKSUdScElFTmxjblJwWm1sallYcHBiMjVsSUVsdWRHVnlibUV3SGhjTk1qTXdNakkyTWpNd01EQXdXaGNOTWpVdwpNakkzTWpJMU9UVTVXakNCcFRFTE1Ba0dBMVVFQmhNQ1NWUXhDekFKQmdOVkJBZ01BazFQTVE4d0RRWURWUVFICkRBWk5iMlJsYm1FeEdUQVhCZ05WQkFvTUVFSlFSVklnUW1GdVkyRWdVeTV3TGtFeE5UQXpCZ05WQkFzTUxGVm0KWmk0Z1UybHpkR1Z0YVNCRWFYQmhjblJwYldWdWRHRnNhU0JsSUVOdmJXMTFibWxqWVhScGIyNXpNU1l3SkFZRApWUVFEREIwcUxtRndjSE11YjJOd0xYQnpZUzB3TVM1blltSndaWEl1Y0hKcGRqQ0NBU0l3RFFZSktvWklodmNOCkFRRUJCUUFEZ2dFUEFEQ0NBUW9DZ2dFQkFLdm1MakhlaUx3L2p0d3hUUFcvUTJzSXdjb3RWSDVEcWxGOEtvMjEKUGExWml6T0dFTXk5R0hhNUEvYXZ6clFXdW10RGNVTExqemdRaDB6N29hWDNxSWJMbklRMHllbDU5Y1ZGZ1ZWTwpVSFpYN29kaUlPMzdkNVhuZ014TnBlcG1jWFFsOEEzV3hkVkRxRnlOT295OGRVT1dCWk1ONGhmLzN3ZUVhWVczCis5S0FlY2JLbS8wUEcrdkFJNms1bm1sUmtYK2w4T0tIVit6US9FMkd3bjByT1JHTml2YjFOV0lYYzZpVENqRjQKL1hmb1c2aE9sTzJWNWpnM0cvSlJ6VUg5UFUwRjJML3dielNmUU15MHF4SzZwOUZGNHdPL3ZqNlNuMGN6NlFzdwpYeWI1Rm01NXpIYitONE04cVFsRU9pc01iYmZhME9va3NCaGVCU05reC9sRWg3VUNBd0VBQWFPQnZqQ0J1ekEvCkJnbGdoa2dCaHZoQ0FRMEVNaFl3UjJWdVpYSmhkR1ZrSUdKNUlIUm9aU0JUWldOMWNtbDBlU0JUWlhKMlpYSWcKWm05eUlIb3ZUMU1nS0ZKQlEwWXBNQ2dHQTFVZEVRUWhNQitDSFNvdVlYQndjeTV2WTNBdGNITmhMVEF4TG1kaQpZbkJsY2k1d2NtbDJNQTRHQTFVZER3RUIvd1FFQXdJRXNEQWRCZ05WSFE0RUZnUVVZVGh5ajVSbVhud1NmdEhKCkE3Y3VJQnEwT0Zvd0h3WURWUjBqQkJnd0ZvQVVxV29Oc1FyVXhpLzI5TERDN1JPNkpPTFpnUEF3RFFZSktvWkkKaHZjTkFRRUxCUUFEZ2dFQkFBRmlJUHZGdHJYaFM0UExpNS94bmZkV1k1UDNnZERTZlJMMDRoWjROd2JZSTVXZwpZRklMVnlwTnVLNVNSTk1hSWw3WkFaUytnUkJoQVRCVStIdjNLN083VHlLaC9UQ2tnOVpHZzI2SVhxc0t5UHZ0CmFPSUx5SDFTZkRiYXhPN3RPclhrTTA3eGd6S0hocUhWWlQ5YXpPUnVrVlJaa01nd0gySGpmdWNTV0ZrTis2eDIKdm1HempBQWlKQ1JGY0JwNnRLYTY0b3RGdFRmY2FNYVZVRlllZkRqZ285b2tnWTRqZ2UwWXlXQkErc1hOUWsydgpVWHhrMm1IYitjSlAxS1I5ayswOEt1YXFKYkJCS0s0RC9tWHkwcCtYODdDYWpZdFRxQ0lHSXk4T3JROTFIYUxiClhYQzlSMWRYV2krMkdrTmI0SkVhKyt0aFVCaVNsQldycHpycURPUT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
  brokerK8sRemoteNamespace: submariner-k8s-broker
  brokerK8sSecret: broker-secret-66fps
  ceIPSecDebug: false
  ceIPSecIKEPort: 500
  ceIPSecNATTPort: 4500
  ceIPSecPSK: 0qZfLf2sx+bVlprOtS7jCuE1wjR9h/HnOfO326ReN63uTFY76bhUTThEqY+WjkLK
  ceIPSecPSKSecret: submariner-ipsec-psk
  clusterCIDR: ""
  clusterID: site1
  connectionHealthCheck:
    enabled: true
    intervalSeconds: 1
    maxPacketLossCount: 5
  debug: true
  namespace: submariner-operator
  natEnabled: true
  repository: quay.io/submariner
  serviceCIDR: ""
  serviceDiscoveryEnabled: true
  version: 0.12.0

Site 2

# oc get submariner submariner -n submariner-operator -o yaml
apiVersion: submariner.io/v1alpha1
kind: Submariner
metadata:
  creationTimestamp: "2024-06-06T13:20:40Z"
  finalizers:
  - controllers.submariner.io/cleanup
  generation: 1
  managedFields:
  - apiVersion: submariner.io/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        .: {}
        f:broker: {}
        f:brokerK8sApiServer: {}
        f:brokerK8sApiServerToken: {}
        f:brokerK8sCA: {}
        f:brokerK8sRemoteNamespace: {}
        f:brokerK8sSecret: {}
        f:ceIPSecDebug: {}
        f:ceIPSecIKEPort: {}
        f:ceIPSecNATTPort: {}
        f:ceIPSecPSK: {}
        f:ceIPSecPSKSecret: {}
        f:clusterCIDR: {}
        f:clusterID: {}
        f:connectionHealthCheck:
          .: {}
          f:enabled: {}
          f:intervalSeconds: {}
          f:maxPacketLossCount: {}
        f:debug: {}
        f:namespace: {}
        f:natEnabled: {}
        f:repository: {}
        f:serviceCIDR: {}
        f:serviceDiscoveryEnabled: {}
        f:version: {}
    manager: subctl
    operation: Update
    time: "2024-06-06T13:20:40Z"
  - apiVersion: submariner.io/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .: {}
          v:"controllers.submariner.io/cleanup": {}
    manager: submariner-operator
    operation: Update
    time: "2024-06-06T13:21:45Z"
  - apiVersion: submariner.io/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:status:
        .: {}
        f:clusterCIDR: {}
        f:clusterID: {}
        f:deploymentInfo: {}
        f:gatewayDaemonSetStatus:
          .: {}
          f:lastResourceVersion: {}
          f:mismatchedContainerImages: {}
          f:nonReadyContainerStates: {}
          f:status:
            .: {}
            f:currentNumberScheduled: {}
            f:desiredNumberScheduled: {}
            f:numberAvailable: {}
            f:numberMisscheduled: {}
            f:numberReady: {}
            f:observedGeneration: {}
            f:updatedNumberScheduled: {}
        f:gateways: {}
        f:globalnetDaemonSetStatus:
          .: {}
          f:mismatchedContainerImages: {}
        f:loadBalancerStatus: {}
        f:natEnabled: {}
        f:networkPlugin: {}
        f:routeAgentDaemonSetStatus:
          .: {}
          f:lastResourceVersion: {}
          f:mismatchedContainerImages: {}
          f:nonReadyContainerStates: {}
          f:status:
            .: {}
            f:currentNumberScheduled: {}
            f:desiredNumberScheduled: {}
            f:numberAvailable: {}
            f:numberMisscheduled: {}
            f:numberReady: {}
            f:observedGeneration: {}
            f:updatedNumberScheduled: {}
        f:serviceCIDR: {}
    manager: submariner-operator
    operation: Update
    subresource: status
    time: "2024-06-06T13:21:54Z"
  name: submariner
  namespace: submariner-operator
  resourceVersion: "958559344"
  uid: 43a6ff7a-b2d4-48c2-84a4-ad1f70ca8e86
spec:
  broker: k8s
  brokerK8sApiServer: api.ocp-psa-01.gbbper.priv:6443
  brokerK8sApiServerToken: eyJhbGciOiJSUzI1NiIsImtpZCI6IngwR2pZY2NWTXJtRlI2T3pxVjFzSHU5OU5SNm9fUkt3V1BMTFJxcnV3LVkifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJzdWJtYXJpbmVyLWs4cy1icm9rZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoiY2x1c3Rlci1zaXRlMi10b2tlbi0yNjRjcCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJjbHVzdGVyLXNpdGUyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiZDM4OGM4ZjMtOGUwZS00NGE3LTlmMDUtMjNkMDFkODU3ZmE5Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OnN1Ym1hcmluZXItazhzLWJyb2tlcjpjbHVzdGVyLXNpdGUyIn0.CP09gdfFSwOR0pk1XGA17Xbg67wOIWd4PqKtFhGUYk3mAvQEjkioM-Vm0UJDCAUqYjH6bzwnWkQxOT8odS8XpMQNd_hj0gHcGXjL5W8DAJzk01ritN-tNHcGJPWC_GxqPT5snAnyDWkL_KwwL4PKl3VEgJhhPbHNgp3fRBh-nbkqjeJ_2gfPr5BySCwAPHOIIZW1V4HTXqvhlFfIKLFiQtPD-PtcqV2P1uy02G7xMVjAMxcHhNFJXIDN8XNrjsCbf28NcoOs7WsjIb0iXI_bPhAQGvfWDWw7kRPIAi7z3OzIcGdrAsVQsw1HcBnAK2CN86b9NybuBGjtVzyUjd9O9QJSxT-OnzKxSj49K33emONX4de1H4hWK0biDccTv_x3gnyTi6HfTJOzfFObBaJVTbconNXfG13odmFiiHTY5z1h3LIfmNGVAi3DJcLBfRfWci2rAbp7niusUsQ0kWlQesIEWvhOG9XlPhp1Gp37R3pEchcQksBKDqUnFVs6o6XTG-Td92i4LdyGJeYslDMY5JHpouVzkrunOZhvwnTsROw6Tb6PbVien2UBGfuW_YGc_9BCgbB2-bxKlBa1NIS-tpr6gk3IjxNLxP5CssTG92888c1yM22USukcpm3nL7lpjCU9Y9Rb3DXynH9n5VjnJ19DpqfVwCU_Q3DrJfF2ilg
  brokerK8sCA: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURNakNDQWhxZ0F3SUJBZ0lJZkRtandjUGF1K1F3RFFZSktvWklodmNOQVFFTEJRQXdOekVTTUJBR0ExVUUKQ3hNSmIzQmxibk5vYVdaME1TRXdId1lEVlFRREV4aHJkV0psTFdGd2FYTmxjblpsY2kxc1lpMXphV2R1WlhJdwpIaGNOTWpNd056SXdNVFV3TVRFMFdoY05Nek13TnpFM01UVXdNVEUwV2pBM01SSXdFQVlEVlFRTEV3bHZjR1Z1CmMyaHBablF4SVRBZkJnTlZCQU1UR0d0MVltVXRZWEJwYzJWeWRtVnlMV3hpTFhOcFoyNWxjakNDQVNJd0RRWUoKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTGNiT0dQbnRaTE1sR2xPQ1FHODFnWCt6dG1FMUFZWgp2RUkybFM4UCtCOGVkSDJCdk9GYkc2dWc1WUhSN1h4RkdlNVllSjZ4Tkw5M01OS05uK1BLcGhrNm5kdlJDTjZtCkF0RFI3WnBBczFYVGZEOHNHOHg3TWFYZTF5RU9SdmtZQXBSLzVOSFFhSnZzUWxqaXY3blQvUWNLTFRKbXpBUUsKS0ZwQ1pLd3NVc1R0NEtsYXVPLzlDWnZpMVZlcTV0NlJhTE1xdjZyREREUC8zcHo4U0NzTlYvR3hIeGhXVWl4dQpVK2RxbmF2RmxGa0xOdys0Qnh1SnFpcUJHNGNwV2p5cjFEMmJBcThvVnoxN1JaTmw2M09VdUxGa3VYdWlsUVBmCmkzSXN1WDQ2WTE0T01icHRzTTltNzZmdElSUWJNU0paZWxqdmlqVG9VKzZ2bFRNak0vUDdkQ0VDQXdFQUFhTkMKTUVBd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3SFFZRFZSME9CQllFRkEwRwpyM2FvSk1UdVpmR3JjZlQrYm94NFZpQkdNQTBHQ1NxR1NJYjNEUUVCQ3dVQUE0SUJBUUE5dkRWRlpFOXhpUzRxCndaQS9DaFRjOG9IMDlCZmlBNTlpNjd0cldLMzZVVGo0SFZiM2ZMOEFXeXBnM2tZSGh2azZ2cUJnRHVmVFpOUjcKU2Y1OS9SY0w1VVRSdTA4bzlETHNvSTlFWUxkMjlRcTdjdHBsUmw1RmZ3UjY5ZjlOL2kwOFlRZDVKLzBLNHA1OAorR1M0WWhTR1d6c2oxUnZ2MUJBTkY2a056MFhGOG1lSVJVWkcyZFBBaFoyVWRxMnBPQ3d2MmFqZHZqWk14ZFBFCkdPMStSbDdBa3hNTEhKbXZDOGJVbEM2U3hsYm1BV2FiaTY5M3VkWjZUQkJwTjRwUkFNZnQvOHFzM3ZsRTlpMzcKajFTWitMTVNQR2xINWlwMUMzbjlDNlUxRlpBQU9vN09uYXJaSk9yZ09ua2h4MVBwM09wTDlYY1I3Q0Y2eE5sMQowbWtYcnY5aQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCi0tLS0tQkVHSU4gQ0VSVElGSUNBVEUtLS0tLQpNSUlEUURDQ0FpaWdBd0lCQWdJSUVsSU52VkpFRVBVd0RRWUpLb1pJaHZjTkFRRUxCUUF3UGpFU01CQUdBMVVFCkN4TUpiM0JsYm5Ob2FXWjBNU2d3SmdZRFZRUURFeDlyZFdKbExXRndhWE5sY25abGNpMXNiMk5oYkdodmMzUXQKYzJsbmJtVnlNQjRYRFRJek1EY3lNREUxTURFeE5Gb1hEVE16TURjeE56RTFNREV4TkZvd1BqRVNNQkFHQTFVRQpDeE1KYjNCbGJuTm9hV1owTVNnd0pnWURWUVFERXg5cmRXSmxMV0Z3YVhObGNuWmxjaTFzYjJOaGJHaHZjM1F0CmMybG5ibVZ5TUlJQklqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUF2Qy82VVNrZEhkT3MKb2NuMFBLdm52RUtEcTZ3bzRkNXJMVGUybG5jam5UVWlRQUFtNkd5YkJRVC9BNTZyb3kxSUpPNGtpWXlscFFBRQpQaEk0dkIzMGdxNjZLS0czdlpkc3dpemt6OTFEVE9wV05ZY3Bmbzh4R3ZzZFZSM1RudVNnREdCVHV3NGNCYzV3Ci9LRXlnS3BtZk1WenJiS0cxbnlNY1daUHhBTkM4V3kyRm1NTSt1eVFFaVRCak9mVVBvWTZkZVVqUXpNZkp5dGsKR1NjMWordEUwejFsWkRZMjJQMW5yTDh3TkR3Y0R3T0gydGcvcHhVV2VkeEQ0SGFLMGwzRjQvOVlxYW56T21oRQo5Q2dRaEtHTGdERzUxUWE0WU1qWUl2bTJqUExvNUpOaU05SjlkZzhKSEdxNm5SM2hkZmI4TkhjY1B4bWRtRzhjClEwWVlYUWMwcXdJREFRQUJvMEl3UURBT0JnTlZIUThCQWY4RUJBTUNBcVF3RHdZRFZSMFRBUUgvQkFVd0F3RUIKL3pBZEJnTlZIUTRFRmdRVVNBZzgvY29iR1lUaitDb0xtMGdaaW9TUnljd3dEUVlKS29aSWh2Y05BUUVMQlFBRApnZ0VCQUtDYkVIN2lKQVN1ajRVcHo0ajd4RVBzY05RS0tOS3RzQ29TbHpuWTBuRTVYUjhwV3VVdmxvTS9xdVVQClUrL3RSWldTaXV3SFdEeUJHY2tubUJhOS92aE1HVzNNcjlCOWxlL0srVHpSQlg0aGF6YWRDbXU2b0p6K1lxWTcKRkQ4elVqYkxZN3BBQ1BFdkpJY3FRWTJTRDdlODd5N0Jnam5UM3lUMzdyd01pcXV6bHI1QmNjLzg3WjRCanNnTAo4MjAyUXVFYml6Q3ZpTnBqYUNyamw4YWtKTG1SRnZZVERiaTk0bzh1YklyVHZ6a1NpbmNhWnQ5eHlibWhySEt4CmhPWjd3Tnh4VzNQRUU5RVZRN1c3RVhLYUhadk1sVWZNUXVrSS84Y05uWDltdm9wSUY5MEVrb3hkd2JQZ00yZ3gKVXN5TllpRENkWENVcGFJWUJFNTUrbWQyb3pJPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCi0tLS0tQkVHSU4gQ0VSVElGSUNBVEUtLS0tLQpNSUlEVERDQ0FqU2dBd0lCQWdJSVc4K0ZZTXhsdklJd0RRWUpLb1pJaHZjTkFRRUxCUUF3UkRFU01CQUdBMVVFCkN4TUpiM0JsYm5Ob2FXWjBNUzR3TEFZRFZRUURFeVZyZFdKbExXRndhWE5sY25abGNpMXpaWEoyYVdObExXNWwKZEhkdmNtc3RjMmxuYm1WeU1CNFhEVEl6TURjeU1ERTFNREV4TkZvWERUTXpNRGN4TnpFMU1ERXhORm93UkRFUwpNQkFHQTFVRUN4TUpiM0JsYm5Ob2FXWjBNUzR3TEFZRFZRUURFeVZyZFdKbExXRndhWE5sY25abGNpMXpaWEoyCmFXTmxMVzVsZEhkdmNtc3RjMmxuYm1WeU1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0MKQVFFQXhLTWdMcmtyc2dSaE4zZEY1TFlBZForeFdmZXNJR0laa0FVZ25tV1Y3azdVbjJCUVkvbUNuekVPTHhibgpDNm56MWFKT3R1SDNoZWg4bE5MVGIxc1JQZkFpaVBIdVBOaDRrZXMxcDdiajVHcXd2MmRNc0tESUw2aHAxbE4xClJEWFNiZUw1ajd6UjFXbUJrL01taUQ2L2wwaWpIZGRjMXpRNmlFQ2p6MXFsbHl6UDM1ZW1VS1dwcG5iQld0S2IKYjJXNGlEdVhBQ0JvbWgzVTBkalZWYitYR3JiQkhaQ21iaWMrRWcvYi8ya3J2WFpZTG93N0QwTjVaRkRkZG84TgpRcFNabTVaOUZYNHE0LzJqMjVZUjAzSGNFd1RwNVI3UXFsWDYxWUZuT2dJWUNQNzB2d0N5aXpVclE4S3dnNGZ6ClZDaGl5Q1pCSXlHYzUrR3hzcUViZXVmL2VRSURBUUFCbzBJd1FEQU9CZ05WSFE4QkFmOEVCQU1DQXFRd0R3WUQKVlIwVEFRSC9CQVV3QXdFQi96QWRCZ05WSFE0RUZnUVVGamJyS3Y0NTFiWmgvaDM1SU9hREtuUnBpd013RFFZSgpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFKK2kzVURUN3Y1cVFpS1gzSW96Qk5MVzBiN3Fncy9tUjJueGFnOWo1RFFuCkZUelRGYjl6dDVUNG5VQ0hwdVRtdjZudWFvUWhrL1BUV2RDV21uSXp5WmVjMERxcCtWWXFCQnFyZGFKUU00b0oKVFdxQTU1WFZGU0s2aVpWS09wNDNETHg2aVhMN1NEWXNiQ000d1I2a3dSbEtib3Z3NjN4SE42QWtWcCtQRzk5dQo3RkR4REU5RG00SW1zd3hhcGFEMnFZTlhkdjRpS1R5cmtHWUg2elM1VmQwSUxtSkFaQjZxMk5sUnpEb2tSY3NTCkovRUJwU3pjVVFuUlRjbkkxM21VMTFqelZRVFRMYktsVXV2d01LU3ZPa0c5bXVsU1p3SUc0RnR5SVd5d1dGbjEKQ093ZUNlaTh5K0FpaE5BYkdET3U1Q0h6WGl3dDVxYko3T3hCeGY2RUh6bz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQotLS0tLUJFR0lOIENFUlRJRklDQVRFLS0tLS0KTUlJRGx6Q0NBbitnQXdJQkFnSUlNRmpCOEN4eWczWXdEUVlKS29aSWh2Y05BUUVMQlFBd1dURlhNRlVHQTFVRQpBd3hPYjNCbGJuTm9hV1owTFd0MVltVXRZWEJwYzJWeWRtVnlMVzl3WlhKaGRHOXlYMnh2WTJGc2FHOXpkQzF5ClpXTnZkbVZ5ZVMxelpYSjJhVzVuTFhOcFoyNWxja0F4TmpnNU9EWTNPRFV6TUI0WERUSXpNRGN5TURFMU5EUXgKTWxvWERUTXpNRGN4TnpFMU5EUXhNMW93V1RGWE1GVUdBMVVFQXd4T2IzQmxibk5vYVdaMExXdDFZbVV0WVhCcApjMlZ5ZG1WeUxXOXdaWEpoZEc5eVgyeHZZMkZzYUc5emRDMXlaV052ZG1WeWVTMXpaWEoyYVc1bkxYTnBaMjVsCmNrQXhOamc1T0RZM09EVXpNSUlCSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQTNCbTUKckxjdS9kcExrczZNcS90TGxvbjJTMFM2aXNzcEZSb1FRZjNwZXg2WDNHN0NLOWpQYldsQ0dWeElpKzdqTW9XYQpHYTYwc3hmeWZ6L0NDMlZpWnJBNUMrUU1GeTJoc29kalFxeEVDNWNVVERmaHRiVzVqYWltY25xUUgrQmRXeHhECmN5c0NWQVNaWFZ4WmRvaVhqTXAwb3A2QWkydzBZekE3UzJDSWFKaTBsTjNBamNFalFRWnNPTEJiMGNrQjFrWHkKNXJMWGI3cTR0WkxQVmtzVm94dncrOUFDNzdxdkNreGwzYWd6TEowZGRjYnFXUjEvS29OdHlRMklpb0pWVmtregoyUHB2ekJ1S3hYUWR0d2s5dWwzWWhwYWIvMEhxM2VjYlpyUGZmY2VVa3RZUjc2cW9RUzhBL0tQNFI1SG5RN3VZClJGUDdQbE9hS3BqdUNqVTVuUUlEQVFBQm8yTXdZVEFPQmdOVkhROEJBZjhFQkFNQ0FxUXdEd1lEVlIwVEFRSC8KQkFVd0F3RUIvekFkQmdOVkhRNEVGZ1FVV3BIQmY2cndMRlpZVXVpYXdQU2VOSHZycDBzd0h3WURWUjBqQkJndwpGb0FVV3BIQmY2cndMRlpZVXVpYXdQU2VOSHZycDBzd0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFLS2o4NW9MClU5a1JLdDlWRVdSNVp4UmdIYzc5TDkrckhYTldxVGpRV2xUcEhrd1B5RE5wLy83OHpUdmNLbkREaEZ1dG0yTE8KMHZCbnlBNlZ3QmdWQS9hR1BnUENGUkxRZk9ndEtHNVFhTW8zUC9ZR3RyN0NHb1NpYUFPblRTZjRybjFlVy9oUAovOGlOZEFzZzVJYU5IMFkrcGM1dEVkUDhUOUNpQmUwUDViNGk3ZFNidXVBVFJOK2U0b2ZvRjV5dEd3RHJ5RjZqCm9MSi9ZUHZFSEtWanhxUWpCYkltdzVVMHJMSWpSZEQ3dlMybmNEWm9YY3Q1djBzeUxuN3JabWd0aWJJRkpFYTAKVng5c2RETkZGSWxWcHBYVXU4RnJ2RHY4TUdsNk9CQUVSZXBCQjhndVpsS0R5TlRNaU1DM3hRTjB0N2wxaXNGagprSWN2ZFFuL1dnV3I2cmc9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0KLS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUVtVENDQTRHZ0F3SUJBZ0lDQzI0d0RRWUpLb1pJaHZjTkFRRUxCUUF3Z2JjeEN6QUpCZ05WQkFZVEFrbFUKTVE4d0RRWURWUVFJRXdaTmIyUmxibUV4RnpBVkJnTlZCQWNURGtObGJuUnlieUJ6WlhKMmFYcHBNUlF3RWdZRApWUVFLRXd0SGNuVndjRzhnUWxCRlVqRVdNQlFHQTFVRUN4TU5RbEJGVWlCVFpYSjJhV05sY3pFVk1CTUdBMVVFCkRCTU1ZMkVnWW5CbGNpQXlNRE00TVRrd053WURWUVFERXpCSGNuVndjRzhnUWxCRlVpQXRJRk5sY25acGVtbHYKSUdScElFTmxjblJwWm1sallYcHBiMjVsSUVsdWRHVnlibUV3SGhjTk1qTXdNakkyTWpNd01EQXdXaGNOTWpVdwpNakkzTWpJMU9UVTVXakNCcFRFTE1Ba0dBMVVFQmhNQ1NWUXhDekFKQmdOVkJBZ01BazFQTVE4d0RRWURWUVFICkRBWk5iMlJsYm1FeEdUQVhCZ05WQkFvTUVFSlFSVklnUW1GdVkyRWdVeTV3TGtFeE5UQXpCZ05WQkFzTUxGVm0KWmk0Z1UybHpkR1Z0YVNCRWFYQmhjblJwYldWdWRHRnNhU0JsSUVOdmJXMTFibWxqWVhScGIyNXpNU1l3SkFZRApWUVFEREIwcUxtRndjSE11YjJOd0xYQnpZUzB3TVM1blltSndaWEl1Y0hKcGRqQ0NBU0l3RFFZSktvWklodmNOCkFRRUJCUUFEZ2dFUEFEQ0NBUW9DZ2dFQkFLdm1MakhlaUx3L2p0d3hUUFcvUTJzSXdjb3RWSDVEcWxGOEtvMjEKUGExWml6T0dFTXk5R0hhNUEvYXZ6clFXdW10RGNVTExqemdRaDB6N29hWDNxSWJMbklRMHllbDU5Y1ZGZ1ZWTwpVSFpYN29kaUlPMzdkNVhuZ014TnBlcG1jWFFsOEEzV3hkVkRxRnlOT295OGRVT1dCWk1ONGhmLzN3ZUVhWVczCis5S0FlY2JLbS8wUEcrdkFJNms1bm1sUmtYK2w4T0tIVit6US9FMkd3bjByT1JHTml2YjFOV0lYYzZpVENqRjQKL1hmb1c2aE9sTzJWNWpnM0cvSlJ6VUg5UFUwRjJML3dielNmUU15MHF4SzZwOUZGNHdPL3ZqNlNuMGN6NlFzdwpYeWI1Rm01NXpIYitONE04cVFsRU9pc01iYmZhME9va3NCaGVCU05reC9sRWg3VUNBd0VBQWFPQnZqQ0J1ekEvCkJnbGdoa2dCaHZoQ0FRMEVNaFl3UjJWdVpYSmhkR1ZrSUdKNUlIUm9aU0JUWldOMWNtbDBlU0JUWlhKMlpYSWcKWm05eUlIb3ZUMU1nS0ZKQlEwWXBNQ2dHQTFVZEVRUWhNQitDSFNvdVlYQndjeTV2WTNBdGNITmhMVEF4TG1kaQpZbkJsY2k1d2NtbDJNQTRHQTFVZER3RUIvd1FFQXdJRXNEQWRCZ05WSFE0RUZnUVVZVGh5ajVSbVhud1NmdEhKCkE3Y3VJQnEwT0Zvd0h3WURWUjBqQkJnd0ZvQVVxV29Oc1FyVXhpLzI5TERDN1JPNkpPTFpnUEF3RFFZSktvWkkKaHZjTkFRRUxCUUFEZ2dFQkFBRmlJUHZGdHJYaFM0UExpNS94bmZkV1k1UDNnZERTZlJMMDRoWjROd2JZSTVXZwpZRklMVnlwTnVLNVNSTk1hSWw3WkFaUytnUkJoQVRCVStIdjNLN083VHlLaC9UQ2tnOVpHZzI2SVhxc0t5UHZ0CmFPSUx5SDFTZkRiYXhPN3RPclhrTTA3eGd6S0hocUhWWlQ5YXpPUnVrVlJaa01nd0gySGpmdWNTV0ZrTis2eDIKdm1HempBQWlKQ1JGY0JwNnRLYTY0b3RGdFRmY2FNYVZVRlllZkRqZ285b2tnWTRqZ2UwWXlXQkErc1hOUWsydgpVWHhrMm1IYitjSlAxS1I5ayswOEt1YXFKYkJCS0s0RC9tWHkwcCtYODdDYWpZdFRxQ0lHSXk4T3JROTFIYUxiClhYQzlSMWRYV2krMkdrTmI0SkVhKyt0aFVCaVNsQldycHpycURPUT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
  brokerK8sRemoteNamespace: submariner-k8s-broker
  brokerK8sSecret: broker-secret-2brrr
  ceIPSecDebug: false
  ceIPSecIKEPort: 500
  ceIPSecNATTPort: 4500
  ceIPSecPSK: 0qZfLf2sx+bVlprOtS7jCuE1wjR9h/HnOfO326ReN63uTFY76bhUTThEqY+WjkLK
  ceIPSecPSKSecret: submariner-ipsec-psk
  clusterCIDR: ""
  clusterID: site2
  connectionHealthCheck:
    enabled: true
    intervalSeconds: 1
    maxPacketLossCount: 5
  debug: true
  namespace: submariner-operator
  natEnabled: true
  repository: quay.io/submariner
  serviceCIDR: ""
  serviceDiscoveryEnabled: true
  version: 0.12.0
status:
  clusterCIDR: 10.132.0.0/14
  clusterID: site2
  deploymentInfo: {}
  gatewayDaemonSetStatus:
    lastResourceVersion: "951307195"
    mismatchedContainerImages: false
    nonReadyContainerStates: []
    status:
      currentNumberScheduled: 3
      desiredNumberScheduled: 3
      numberAvailable: 3
      numberMisscheduled: 0
      numberReady: 3
      observedGeneration: 1
      updatedNumberScheduled: 3
  gateways:
  - connections:
    - endpoint:
        backend: libreswan
        backend_config:
          natt-discovery-port: "4490"
          preferred-server: "false"
          public-ip: dns:control-1-ru4.ocp-psa-01.gbbper.priv
          udp-port: "4500"
        cable_name: submariner-cable-site1-192-168-54-32
        cluster_id: site1
        healthCheckIP: 10.129.0.2
        hostname: control-1-ru4.ocp-psa-01.gbbper.priv
        nat_enabled: true
        private_ip: 192.168.54.32
        public_ip: 192.168.54.32
        subnets:
        - 172.30.0.0/16
        - 10.128.0.0/14
      latencyRTT:
        average: 1.181959ms
        last: 1.103177ms
        max: 66.129596ms
        min: 924.003µs
        stdDev: 704.858µs
      status: connected
      statusMessage: ""
      usingIP: 192.168.54.32
    haStatus: active
    localEndpoint:
      backend: libreswan
      backend_config:
        natt-discovery-port: "4490"
        preferred-server: "false"
        public-ip: dns:control-1-ru2.ocp-psb-01.gbbper.priv
        udp-port: "4500"
      cable_name: submariner-cable-site2-192-168-126-30
      cluster_id: site2
      healthCheckIP: 10.133.0.2
      hostname: control-1-ru2.ocp-psb-01.gbbper.priv
      nat_enabled: true
      private_ip: 192.168.126.30
      public_ip: 192.168.126.30
      subnets:
      - 172.31.0.0/16
      - 10.132.0.0/14
    statusFailure: ""
    version: release-0.12-621cbce
  - connections: []
    haStatus: passive
    localEndpoint:
      backend: libreswan
      backend_config:
        natt-discovery-port: "4490"
        preferred-server: "false"
        public-ip: dns:control-1-ru3.ocp-psb-01.gbbper.priv
        udp-port: "4500"
      cable_name: submariner-cable-site2-192-168-126-31
      cluster_id: site2
      healthCheckIP: 10.132.0.2
      hostname: control-1-ru3.ocp-psb-01.gbbper.priv
      nat_enabled: true
      private_ip: 192.168.126.31
      public_ip: 192.168.126.31
      subnets:
      - 172.31.0.0/16
      - 10.132.0.0/14
    statusFailure: ""
    version: release-0.12-621cbce
  - connections: []
    haStatus: passive
    localEndpoint:
      backend: libreswan
      backend_config:
        natt-discovery-port: "4490"
        preferred-server: "false"
        public-ip: dns:control-1-ru4.ocp-psb-01.gbbper.priv
        udp-port: "4500"
      cable_name: submariner-cable-site2-192-168-126-32
      cluster_id: site2
      healthCheckIP: 10.134.0.2
      hostname: control-1-ru4.ocp-psb-01.gbbper.priv
      nat_enabled: true
      private_ip: 192.168.126.32
      public_ip: 192.168.126.32
      subnets:
      - 172.31.0.0/16
      - 10.132.0.0/14
    statusFailure: ""
    version: release-0.12-621cbce
  globalnetDaemonSetStatus:
    mismatchedContainerImages: false
  loadBalancerStatus: {}
  natEnabled: true
  networkPlugin: OVNKubernetes
  routeAgentDaemonSetStatus:
    lastResourceVersion: "945546125"
    mismatchedContainerImages: false
    nonReadyContainerStates: []
    status:
      currentNumberScheduled: 12
      desiredNumberScheduled: 12
      numberAvailable: 12
      numberMisscheduled: 0
      numberReady: 12
      observedGeneration: 1
      updatedNumberScheduled: 12
  serviceCIDR: 172.31.0.0/16
  1. oc get network cluster -o yaml
    Site 1
# oc get network cluster -o yaml
apiVersion: config.openshift.io/v1
kind: Network
metadata:
  creationTimestamp: "2023-07-20T15:08:25Z"
  generation: 2
  managedFields:
  - apiVersion: config.openshift.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        .: {}
        f:clusterNetwork: {}
        f:externalIP:
          .: {}
          f:policy: {}
        f:networkType: {}
        f:serviceNetwork: {}
      f:status: {}
    manager: cluster-bootstrap
    operation: Update
    time: "2023-07-20T15:08:25Z"
  - apiVersion: config.openshift.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:status:
        f:clusterNetwork: {}
        f:clusterNetworkMTU: {}
        f:networkType: {}
        f:serviceNetwork: {}
    manager: cluster-network-operator
    operation: Update
    time: "2023-07-20T15:40:50Z"
  name: cluster
  resourceVersion: "6285"
  uid: 42ed0cab-58a1-4229-8d8a-78ddf7430518
spec:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  externalIP:
    policy: {}
  networkType: OVNKubernetes
  serviceNetwork:
  - 172.30.0.0/16
status:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  clusterNetworkMTU: 1400
  networkType: OVNKubernetes
  serviceNetwork:
  - 172.30.0.0/16

Site 2

#  oc get network cluster -o yaml
apiVersion: config.openshift.io/v1
kind: Network
metadata:
  creationTimestamp: "2023-08-16T08:57:56Z"
  generation: 2
  managedFields:
  - apiVersion: config.openshift.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        .: {}
        f:clusterNetwork: {}
        f:externalIP:
          .: {}
          f:policy: {}
        f:networkType: {}
        f:serviceNetwork: {}
      f:status: {}
    manager: cluster-bootstrap
    operation: Update
    time: "2023-08-16T08:57:56Z"
  - apiVersion: config.openshift.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:status:
        f:clusterNetwork: {}
        f:clusterNetworkMTU: {}
        f:networkType: {}
        f:serviceNetwork: {}
    manager: cluster-network-operator
    operation: Update
    time: "2023-08-16T09:31:10Z"
  name: cluster
  resourceVersion: "5877"
  uid: 7b575eb7-3655-4244-a157-71a87077b829
spec:
  clusterNetwork:
  - cidr: 10.132.0.0/14
    hostPrefix: 23
  externalIP:
    policy: {}
  networkType: OVNKubernetes
  serviceNetwork:
  - 172.31.0.0/16
status:
  clusterNetwork:
  - cidr: 10.132.0.0/14
    hostPrefix: 23
  clusterNetworkMTU: 1400
  networkType: OVNKubernetes
  serviceNetwork:
  - 172.31.0.0/16

@BhavaniYalamanchili
Copy link
Author

@dfarrell07
If the submariner needs to be upgraded, Can you point out which submariner version is compatible with OCP 4.10?

@yboaron
Copy link
Contributor

yboaron commented Jun 13, 2024

@BhavaniYalamanchili

Here are the outputs:

oc get pods -A -1 name=ovnkube-db
No resources found

Did you mean oc get pods -A -l name=ovnkube-db ?

@BhavaniYalamanchili
Copy link
Author

BhavaniYalamanchili commented Jun 13, 2024

@yboaron
Sorry typo, it is -l not -1

Did you mean oc get pods -A -l name=ovnkube-db ?

Yes

@tpantelis
Copy link
Contributor

@dfarrell07 If the submariner needs to be upgraded, Can you point out which submariner version is compatible with OCP 4.10?

You had created #2955 a while back where you were using Submariner 0.16.x and OCP 4.15 so I'm curious why you would now be using much older versions of each (which are no longer supported)....

@yboaron
Copy link
Contributor

yboaron commented Jun 13, 2024

A.

oc get pods -A -1 name=ovnkube-db
No resources found

Hmm, that's weird, b/c if no pod with ovnkube-db label is found Submariner shouldn't recognize CNI plugin as OVN-K8S, and OVN-K8S was detected successfully for site2

B. What version of subctl do you use ?

C. Also think it is better to upgrade Submariner, for OCP 4.10 you can upgrade to Submariner 0.14.

D. If you decide to stay with version 0.12, and still hit this issue please upload subctl gather from both clusters.

@BhavaniYalamanchili
Copy link
Author

BhavaniYalamanchili commented Jun 13, 2024

@tpantelis This is a different cluster setup

@yboaron
B. 0.12.0 subctl only is being used

C. Any version of 0.14.x would work on OCP 4.10?

D. Sure, will try to get the subctl gather from both clusters, it might take some time. Meanwhile, I have found some error log statements in submariner-operator logs of Site 1, Please check this out.

[90m2024-06-06T13:24:47.742Z[0m [32mINF[0m ..e-arguments/main.go:174 cmd                  Could not generate and serve custom resource metrics [36merror=[0m[31m"error initializing metrics: discovering resource information failed for Submariner in submariner.io/v1alpha1: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request"[0m

[90m2024-06-06T13:25:22.495Z[0m [32mINF[0m ..e-arguments/main.go:229 cmd                  Could not create ServiceMonitor object [36merror=[0m[31m"unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request"[0m

[90m2024-06-06T13:25:40.455Z[0m [1m[31mERR[0m[0m ..oller/controller.go:267 ..mariner-controller Reconciler error [36merror=[0m[31m"unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request"[0m [36mname=[0msubmariner [36mnamespace=[0msubmariner-operator [36mreconciler group=[0msubmariner.io [36mreconciler kind=[0mSubmariner

@yboaron
Copy link
Contributor

yboaron commented Jun 16, 2024

C. Any version of 0.14.x would work on OCP 4.10?

Yep.

@BhavaniYalamanchili
Copy link
Author

@yboaron There is a concern about sharing all the logs. Could you let me know if you want any specific logs to look for the exact error?

@yboaron
Copy link
Contributor

yboaron commented Jun 17, 2024

Please share the output of oc get submariner submariner -n submariner-operator -o yaml , from site1 , couldn't see the status section in the ^^ you attached above.
and also submariner-operator pod logs from both clusters

@BhavaniYalamanchili
Copy link
Author

@yboaron

The output of the command oc get submariner submariner -n submariner-operator -o yaml is displayed till the spec only the status section is not displayed, I wonder why
In the yaml file that is collected also its the same

Here are the submariner-operator pod logs you asked
Please change the the extension to zip when downloaded
SIte1 logs.txt
SIte2 logs.txt

@yboaron
Copy link
Contributor

yboaron commented Jun 19, 2024

According to the site1 logs, submariner-operator failed to reconcile due to [1] error, and therefore does not update submariner.status section.

Please address this issue and let us know how it goes.

[1]

unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request

@tpantelis
Copy link
Contributor

unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request

This looks like an incompatibility with the K8s version your using, ie the K8s version is newer than what's supported by Submariner 0.12. As mentioned earlier, 0.12 is no longer maintained so I strongly suggest upgrading Submariner.

@BhavaniYalamanchili
Copy link
Author

In the past when we had an issue with Submariner 0.12.0 and OCP 4.10, then it was said by your team that the OCP version must be 4.11+ for the submariner 0.13.
#1978 (comment)
Also pointed to the third bullet of the doc https://github.com/submariner-io/releases/releases/tag/v0.13.0
So will the submariner 0.14.6 work on OCP 4.10?

Also in the same ticket, it was mentioned that the Submariner does not support OVN on OCP 4.9 and 4.10
#1978 (comment)

@tpantelis
Copy link
Contributor

You're better off upgrading both Submariner and OCP to supported versions.

@BhavaniYalamanchili
Copy link
Author

BhavaniYalamanchili commented Jun 20, 2024

@yboaron @tpantelis

We upgraded the Submariner to 0.14.6 version, and even after the upgrade its the same error message we are seeing and some add on errors

Site 1

./subctl-v0.14.6-linux-amd64 show all --kubeconfig=/tmp/site-1-kubeconfig
I0620 11:47:51.523338 1183225 request.go:601] Waited for 1.010687181s due to client-side throttling, not priority and fairness, request: GET:https://api.ocp-psa-01.gbbper.priv:6443/apis/ibmcpcs.ibm.com/v1?timeout=32s
I0620 11:48:01.523343 1183225 request.go:601] Waited for 10.921594654s due to client-side throttling, not priority and fairness, request: GET:https://api.ocp-psa-01.gbbper.priv:6443/apis/machineconfiguration.openshift.io/v1?timeout=32s
Cluster "site1"
I0620 11:48:11.542570 1183225 request.go:601] Waited for 5.340290374s due to client-side throttling, not priority and fairness, request: GET:https://api.ocp-psa-01.gbbper.priv:6443/apis/satellite.isf.ibm.com/v1?timeout=32s
 ✓ Detecting broker(s)
NAMESPACE               NAME                COMPONENTS                        GLOBALNET   GLOBALNET CIDR   DEFAULT GLOBALNET SIZE   DEFAULT DOMAINS
submariner-k8s-broker   submariner-broker   service-discovery, connectivity   no          242.0.0.0/8      65536

 ✓ Showing Connections
GATEWAY                          CLUSTER   REMOTE IP        NAT   CABLE DRIVER   SUBNETS                        STATUS   RTT avg.
control-1-ru4.ocp-psb-01.gbbpe   site2     192.168.126.32   no    libreswan      172.31.0.0/16, 10.132.0.0/14   error    0s

 ✓ Showing Endpoints
CLUSTER   ENDPOINT IP      PUBLIC IP        CABLE DRIVER   TYPE
site1     192.168.54.30    192.168.54.30    libreswan      local
site2     192.168.126.32   192.168.126.32   libreswan      remote
site1     192.168.54.31    192.168.54.31    libreswan      local
site1     192.168.54.32    192.168.54.32    libreswan      local

 ✓ Showing Gateways
NODE                             HA STATUS   SUMMARY
control-1-ru2.ocp-psa-01.gbbpe   active      0 connections out of 1 are established
control-1-ru3.ocp-psa-01.gbbpe   passive     There are no connections
control-1-ru4.ocp-psa-01.gbbpe   passive     There are no connections

 ✓ Showing Network details
    Discovered network details via Submariner:
        Network plugin:
        Service CIDRs:   []
        Cluster CIDRs:   []

 ✓ Showing versions
COMPONENT             REPOSITORY           VERSION
submariner-gateway    quay.io/submariner   0.14.6
submariner-operator   quay.io/submariner   0.14.6
sh-4.4$ ./subctl-v0.14.6-linux-amd64 show all --kubeconfig=/tmp/site-1-kubeconfig [root@bstcp4s-p-sl-01 subctl-v0.14.6]# oc rsh isf-metrodr-operator-controller-manager-c88b64864-5m9bb
error: You must be logged in to the server (Unauthorized)
[root@bstcp4s-p-sl-01 subctl-v0.14.6]# oc login https://api.ocp-psb-01.gbbper.priv:6443/ -u kubeadmin -p 43aTz-Ipzat-AzVqt-4YhpF --insecure-skip-tls-verify=true
Login successful.

You have access to 76 projects, the list has been suppressed. You can list all projects with ' projects'

Using project "ibm-spectrum-fusion-ns".
[root@bstcp4s-p-sl-01 subctl-v0.14.6]# oc rsh isf-metrodr-operator-controller-manager-c88b64864-5m9bb                                                        Defaulting container name to manager.
Use 'oc describe pod/isf-metrodr-operator-controller-manager-c88b64864-5m9bb -n ibm-spectrum-fusion-ns' to see all of the containers in this pod.
sh-4.4$ cd /tmp/subctl_new/
sh-4.4$ ./subctl-v0.14.6-linux-amd64 diagnose all --kubeconfig=/tmp/site-1-kubeconfig
Cluster "site1"
I0620 11:49:27.218182 1183270 request.go:601] Waited for 1.008715728s due to client-side throttling, not priority and fairness, request: GET:https://api.ocp-psa-01.gbbper.priv:6443/apis/spp-data-protection.isf.ibm.com/v1alpha1?timeout=32s
I0620 11:49:37.417620 1183270 request.go:601] Waited for 11.208065595s due to client-side throttling, not priority and fairness, request: GET:https://api.ocp-psa-01.gbbper.priv:6443/apis/authorization.openshift.io/v1?timeout=32s
 ✓ Checking Submariner support for the Kubernetes version
 ✓ Kubernetes version "v1.23.12+8a6bfe4" is supported

 ✗ Checking Submariner support for the CNI network plugin
 ✗ The detected CNI plugin ("") is not supported by Submariner. Supported plugins: [generic canal-flannel weave-net OpenShiftSDN OVNKubernetes calico kindnet]
 ✗ Checking gateway connections
 ✗ Connection to cluster "site2" is not established. Connection details:
{
  "status": "error",
  "statusMessage": "Failed to successfully ping the remote endpoint IP \"10.134.0.2\"",
  "endpoint": {
    "cluster_id": "site2",
    "cable_name": "submariner-cable-site2-192-168-126-32",
    "healthCheckIP": "10.134.0.2",
    "hostname": "control-1-ru4.ocp-psb-01.gbbper.priv",
    "subnets": [
      "172.31.0.0/16",
      "10.132.0.0/14"
    ],
    "private_ip": "192.168.126.32",
    "public_ip": "192.168.126.32",
    "nat_enabled": true,
    "backend": "libreswan",
    "backend_config": {
      "natt-discovery-port": "4490",
      "preferred-server": "false",
      "public-ip": "dns:control-1-ru4.ocp-psb-01.gbbper.priv",
      "udp-port": "4500"
    }
  },
  "usingIP": "192.168.126.32",
  "latencyRTT": {
    "last": "0s",
    "min": "0s",
    "average": "0s",
    "max": "0s",
    "stdDev": "0s"
  }
}
 ✓ Non-Globalnet deployment detected - checking if cluster CIDRs overlap
 ✓ Clusters do not have overlapping CIDRs
 ✗ Checking Submariner pods
 ✗ Error obtaining Daemonset "submariner-routeagent": daemonsets.apps "submariner-routeagent" not found
 ✗ Error obtaining Deployment "submariner-lighthouse-agent": deployments.apps "submariner-lighthouse-agent" not found
 ✗ Error obtaining Deployment "submariner-lighthouse-coredns": deployments.apps "submariner-lighthouse-coredns" not found
 ✗ Error obtaining Daemonset "submariner-metrics-proxy": daemonsets.apps "submariner-metrics-proxy" not found
 ✓ Checking Submariner support for the kube-proxy mode
 ✓ The kube-proxy mode is supported
 ✗ Checking the firewall configuration to determine if intra-cluster VXLAN traffic is allowed
 ✗ The tcpdump output from the sniffer pod does not contain the expected remote endpoint IP 172.31.0.0. Please check that your firewall configuration allows UDP/4800 traffic.
 ✓ Globalnet is not installed - skipping

 ⚠ Service discovery is not installed

Skipping inter-cluster firewall check as it requires two kubeconfigs. Please run "subctl diagnose firewall inter-cluster" command manually.

subctl version: v0.14.6

Site 2

sh-4.4$ ./subctl-v0.14.6-linux-amd64 show all --kubeconfig=/tmp/site-2-kubeconfig
Cluster "local-config"
 ✓ Detecting broker(s)
 ✓ No brokers found

 ✓ Showing Connections
GATEWAY                          CLUSTER   REMOTE IP       NAT   CABLE DRIVER   SUBNETS                        STATUS      RTT avg.
control-1-ru2.ocp-psa-01.gbbpe   site1     192.168.54.30   no    libreswan      172.30.0.0/16, 10.128.0.0/14   connected   1.400162ms

 ✓ Showing Endpoints
CLUSTER   ENDPOINT IP      PUBLIC IP        CABLE DRIVER   TYPE
site2     192.168.126.30   192.168.126.30   libreswan      local
site2     192.168.126.31   192.168.126.31   libreswan      local
site2     192.168.126.32   192.168.126.32   libreswan      local
site1     192.168.54.30    192.168.54.30    libreswan      remote

 ✓ Showing Gateways
NODE                             HA STATUS   SUMMARY
control-1-ru2.ocp-psb-01.gbbpe   passive     There are no connections
control-1-ru3.ocp-psb-01.gbbpe   passive     There are no connections
control-1-ru4.ocp-psb-01.gbbpe   active      All connections (1) are established

 ✓ Showing Network details
    Discovered network details via Submariner:
        Network plugin:  OVNKubernetes
        Service CIDRs:   [172.31.0.0/16]
        Cluster CIDRs:   [10.132.0.0/14]

 ✓ Showing versions
COMPONENT                       REPOSITORY           VERSION
submariner-gateway              quay.io/submariner   0.14.6
submariner-routeagent           quay.io/submariner   0.14.6
submariner-operator             quay.io/submariner   0.14.6
submariner-lighthouse-agent     quay.io/submariner   0.14.6
submariner-lighthouse-coredns   quay.io/submariner   0.14.6

On site2 there is a difference from previous version output

sh-4.4$ /tmp/subctl_new/subctl-v0.14.6-linux-amd64 diagnose all --kubeconfig=/tmp/site-2-kubeconfig
Cluster "local-config"
 ✓ Checking Submariner support for the Kubernetes version
 ✓ Kubernetes version "v1.23.12+8a6bfe4" is supported

 ✓ Checking Submariner support for the CNI network plugin
 ✓ The detected CNI network plugin ("OVNKubernetes") is supported
 ✗ Checking OVN version
 ✗ The ovn-nb database version 5.35.1 is less than the minimum supported version 6.1.0
 ✓ Checking gateway connections
 ✓ All connections are established
 ✓ Non-Globalnet deployment detected - checking if cluster CIDRs overlap
 ✓ Clusters do not have overlapping CIDRs
 ✗ Checking Submariner pods
 ✗ The desired number of replicas for Deployment "submariner-networkplugin-syncer" (1) does not match the actual number running (0)
 ⚠ Pod "submariner-networkplugin-syncer-547fff98b6-qbz8v" has restarted 13 times
 ✓ Checking Submariner support for the kube-proxy mode
 ✓ The kube-proxy mode is supported
 ✓ Checking the firewall configuration to determine if intra-cluster VXLAN traffic is allowed
 ✓ This check is not necessary for the OVNKubernetes CNI plugin
 ✓ The firewall configuration allows intra-cluster VXLAN traffic
 ✓ Globalnet is not installed - skipping

 ✓ Checking if services have been exported properly
 ✓ All services have been exported properly

Skipping inter-cluster firewall check as it requires two kubeconfigs. Please run "subctl diagnose firewall inter-cluster" command manually.

subctl version: v0.14.6

@yboaron
Copy link
Contributor

yboaron commented Jun 23, 2024

First of all sorry I wasn't clear in my previous answer, Submariner 0.14.x can work with OCP 4.10 but with CNI other than OVN-K.

As suggested, please upgrade also OCP to version 4.11+, as
OVN-K requires OVN NorthBound DB version 6.1.0+, available with OCP 4.11.0+

@BhavaniYalamanchili
Copy link
Author

BhavaniYalamanchili commented Jun 24, 2024

@yboaron we have some concerns about updating the OCP 4.10 to 4.11, we want to get the submariner issue fixed through which a component that is dependent on the submariner will be fixed and then only we can proceed to the OCP upgrade.

@tpantelis said that it looks like an incompatibility issue

This looks like an incompatibility with the K8s version your using, ie the K8s version is newer than what's supported by Submariner 0.12.

So, which K8s version is compatible with Submariner 0.12.0?

Another point I see is that the Submariner 0.12. and also 0.14. versions are trying to fetch a pod that has the label name=ovnkube-db on it

And we don't have any pods like that on both sites,

# oc get pods -A -l name=ovnkube-db

# oc get pods -n openshift-ovn-kubernetes -l name=ovnkube-db
No resources found in openshift-ovn-kubernetes namespace.

But the Site 2 is able to detect the plugin
Is there any other way that Site2 is able to detect the CNI plugin?

Here I want to mention a point on what happened before this issue
There was an issue with Site2, it wasn't accessible neither via oc cli nor via web-console) when we approached RedHat, the RedHat support team recreated the OVN database on Site2 and then we observed this issue with the submariner.
Do you think the recreation of the OVN database on Site 2 is in any way relatable to this issue?

@yboaron
Copy link
Contributor

yboaron commented Jun 26, 2024

Well, I'm afraid it's a chicken and egg problem here, Submariner doesn't support OCP 4.10 with OVN-K as cni and you want to upgrade OCP after Submariner issue resolved.

I think the best thing would be to uninstall Submariner, upgrade OCP, reinstall Submariner (of course upgrading submariner version).

Maybe you can try resolve Submariner issue before upgrading OCP using the following workaround:

  • Change OCP CNI to OpenshiftSDN
  • Address [1] error on site1, after short search on Google I found these [2] links
  • Reinstall Submariner

[1]

unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request

[2]
https://cloud.ibm.com/docs/containers?topic=containers-debug_metrics_server
https://pet2cattle.com/2021/05/unable-to-retrieve-the-complete-list-of-server-apis

Copy link
Contributor

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further
activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale label Oct 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Backlog
Development

No branches or pull requests

4 participants