All along the tests, the RG status (isSource always remained true for source), SyncIQ policy on PowerScale arrays and data on the source & target PVs were continuously monitored. [root@master-1-PPRvj6ITxSpjq repctl]# ./repctl get rg [2022-12-05 12:35:00] INFO listing replication groups [2022-12-05 12:35:00] INFO Cluster: kubernetes2 +-----+ | RG | +-----+ Name State rClusterID Driver RemoteRG IsSource LinkState rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 Ready sl-k8s csi-isilon.dellemc.com rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 false SYNCHRONIZED [2022-12-05 12:35:00] INFO [2022-12-05 12:35:00] INFO Cluster: sl-k8s +-----+ | RG | +-----+ Name State rClusterID Driver RemoteRG IsSource LinkState rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 Ready kubernetes2 csi-isilon.dellemc.com rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 true SYNCHRONIZED [root@master-1-PPRvj6ITxSpjq repctl]# ./repctl --rg rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 exec -a suspend [2022-12-05 12:36:32] INFO RG (rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52) on cluster (sl-k8s), successfully updated with action: (SUSPEND) [2022-12-05 12:45:58] INFO Cluster: kubernetes2 +-----+ | RG | +-----+ Name State rClusterID Driver RemoteRG IsSource LinkState rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 Ready sl-k8s csi-isilon.dellemc.com rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 false SUSPENDED [2022-12-05 12:45:58] INFO [2022-12-05 12:45:58] INFO Cluster: sl-k8s +-----+ | RG | +-----+ Name State rClusterID Driver RemoteRG IsSource LinkState rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 Ready kubernetes2 csi-isilon.dellemc.com rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 true SUSPENDED [root@master-1-PPRvj6ITxSpjq repctl]# ./repctl --rg rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 exec -a resume [2022-12-05 12:49:35] INFO RG (rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52) on cluster (sl-k8s), successfully updated with action: (RESUME) [2022-12-05 12:49:53] INFO Cluster: kubernetes2 +-----+ | RG | +-----+ Name State rClusterID Driver RemoteRG IsSource LinkState rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 Ready sl-k8s csi-isilon.dellemc.com rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 false SYNCHRONIZED [2022-12-05 12:49:53] INFO [2022-12-05 12:49:53] INFO Cluster: sl-k8s +-----+ | RG | +-----+ Name State rClusterID Driver RemoteRG IsSource LinkState rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 Ready kubernetes2 csi-isilon.dellemc.com rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 true SYNCHRONIZED [root@master-1-PPRvj6ITxSpjq repctl]# ./repctl --rg rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 exec -a sync [2022-12-05 13:00:00] INFO RG (rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52) on cluster (sl-k8s), successfully updated with action: (SYNC) [2022-12-05 13:00:03] INFO Cluster: kubernetes2 +-----+ | RG | +-----+ Name State rClusterID Driver RemoteRG IsSource LinkState rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 Ready sl-k8s csi-isilon.dellemc.com rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 false SYNCHRONIZED [2022-12-05 13:00:03] INFO [2022-12-05 13:00:03] INFO Cluster: sl-k8s +-----+ | RG | +-----+ Name State rClusterID Driver RemoteRG IsSource LinkState rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 SYNC_IN_PROGRESS kubernetes2 csi-isilon.dellemc.com rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 true SYNCHRONIZED [2022-12-05 13:00:34] INFO Cluster: kubernetes2 +-----+ | RG | +-----+ Name State rClusterID Driver RemoteRG IsSource LinkState rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 Ready sl-k8s csi-isilon.dellemc.com rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 false SYNCHRONIZED [2022-12-05 13:00:34] INFO [2022-12-05 13:00:34] INFO Cluster: sl-k8s +-----+ | RG | +-----+ Name State rClusterID Driver RemoteRG IsSource LinkState rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 Ready kubernetes2 csi-isilon.dellemc.com rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 true SYNCHRONIZED [root@master-1-PPRvj6ITxSpjq repctl]# ./repctl --rg rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 failover --target kubernetes2 [2022-12-05 13:02:22] INFO RG (rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52), successfully updated with action: failover [2022-12-05 13:02:24] INFO Cluster: kubernetes2 +-----+ | RG | +-----+ Name State rClusterID Driver RemoteRG IsSource LinkState rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 Ready sl-k8s csi-isilon.dellemc.com rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 false SYNCHRONIZED [2022-12-05 13:02:24] INFO [2022-12-05 13:02:24] INFO Cluster: sl-k8s +-----+ | RG | +-----+ Name State rClusterID Driver RemoteRG IsSource LinkState rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 FAILOVER_REMOTE_IN_PROGRESS kubernetes2 csi-isilon.dellemc.com rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 true SYNCHRONIZED [2022-12-05 13:04:11] INFO Cluster: kubernetes2 +-----+ | RG | +-----+ Name State rClusterID Driver RemoteRG IsSource LinkState rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 Ready sl-k8s csi-isilon.dellemc.com rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 false FAILEDOVER [2022-12-05 13:04:11] INFO [2022-12-05 13:04:11] INFO Cluster: sl-k8s +-----+ | RG | +-----+ Name State rClusterID Driver RemoteRG IsSource LinkState rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 Ready kubernetes2 csi-isilon.dellemc.com rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 true FAILEDOVER [root@master-1-PPRvj6ITxSpjq repctl]# ./repctl --rg rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 failback --target sl-k8s --discard [2022-12-05 13:07:02] INFO RG (rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52), successfully updated with action: failback [2022-12-05 13:07:07] INFO Cluster: kubernetes2 +-----+ | RG | +-----+ Name State rClusterID Driver RemoteRG IsSource LinkState rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 Ready sl-k8s csi-isilon.dellemc.com rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 false FAILEDOVER [2022-12-05 13:07:07] INFO [2022-12-05 13:07:07] INFO Cluster: sl-k8s +-----+ | RG | +-----+ Name State rClusterID Driver RemoteRG IsSource LinkState rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 ACTION_FAILBACK_DISCARD_CHANGES_LOCAL_IN_PROGRESS kubernetes2 csi-isilon.dellemc.com rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 true FAILEDOVER [2022-12-05 13:08:32] INFO Cluster: kubernetes2 +-----+ | RG | +-----+ Name State rClusterID Driver RemoteRG IsSource LinkState rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 Ready sl-k8s csi-isilon.dellemc.com rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 false SYNCHRONIZED [2022-12-05 13:08:32] INFO [2022-12-05 13:08:32] INFO Cluster: sl-k8s +-----+ | RG | +-----+ Name State rClusterID Driver RemoteRG IsSource LinkState rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 Ready kubernetes2 csi-isilon.dellemc.com rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 true SYNCHRONIZED [root@master-1-PPRvj6ITxSpjq repctl]# ./repctl --rg rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 failover --target kubernetes2 --unplanned [2022-12-05 13:17:12] INFO RG (rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52), successfully updated with action: unplanned failover [2022-12-05 13:17:15] INFO Cluster: kubernetes2 +-----+ | RG | +-----+ Name State rClusterID Driver RemoteRG IsSource LinkState rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 UNPLANNED_FAILOVER_LOCAL_IN_PROGRESS sl-k8s csi-isilon.dellemc.com rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 false SYNCHRONIZED [2022-12-05 13:17:15] INFO [2022-12-05 13:17:15] INFO Cluster: sl-k8s +-----+ | RG | +-----+ Name State rClusterID Driver RemoteRG IsSource LinkState rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 Ready kubernetes2 csi-isilon.dellemc.com rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 true SYNCHRONIZED [2022-12-05 13:18:43] INFO Cluster: kubernetes2 +-----+ | RG | +-----+ Name State rClusterID Driver RemoteRG IsSource LinkState rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 Ready sl-k8s csi-isilon.dellemc.com rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 false FAILEDOVER [2022-12-05 13:18:43] INFO [2022-12-05 13:18:43] INFO Cluster: sl-k8s +-----+ | RG | +-----+ Name State rClusterID Driver RemoteRG IsSource LinkState rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 Ready kubernetes2 csi-isilon.dellemc.com rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 true FAILEDOVER [root@master-1-PPRvj6ITxSpjq repctl]# ./repctl --rg rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 failback --target sl-k8s [2022-12-05 13:19:33] INFO RG (rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52), successfully updated with action: failback [2022-12-05 13:19:37] INFO Cluster: kubernetes2 +-----+ | RG | +-----+ Name State rClusterID Driver RemoteRG IsSource LinkState rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 Ready sl-k8s csi-isilon.dellemc.com rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 false FAILEDOVER [2022-12-05 13:19:37] INFO [2022-12-05 13:19:37] INFO Cluster: sl-k8s +-----+ | RG | +-----+ Name State rClusterID Driver RemoteRG IsSource LinkState rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 FAILBACK_LOCAL_IN_PROGRESS kubernetes2 csi-isilon.dellemc.com rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 true FAILEDOVER [2022-12-05 13:22:06] INFO Cluster: kubernetes2 +-----+ | RG | +-----+ Name State rClusterID Driver RemoteRG IsSource LinkState rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 Ready sl-k8s csi-isilon.dellemc.com rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 false SYNCHRONIZED [2022-12-05 13:22:06] INFO [2022-12-05 13:22:06] INFO Cluster: sl-k8s +-----+ | RG | +-----+ Name State rClusterID Driver RemoteRG IsSource LinkState rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 Ready kubernetes2 csi-isilon.dellemc.com rg-18e5843b-3c36-40ae-ad7d-e6a7cc5faf52 true SYNCHRONIZED ----reprotect----- [2022-12-07 10:21:11] INFO Cluster: kubernetes2 +-----+ | RG | +-----+ Name State rClusterID Driver RemoteRG IsSource LinkState rg-e0e401e1-54dc-4983-9353-183d19b68cf6 Ready sl-k8s csi-isilon.dellemc.com rg-e0e401e1-54dc-4983-9353-183d19b68cf6 false FAILEDOVER [2022-12-07 10:21:11] INFO [2022-12-07 10:21:11] INFO Cluster: sl-k8s +-----+ | RG | +-----+ Name State rClusterID Driver RemoteRG IsSource LinkState rg-e0e401e1-54dc-4983-9353-183d19b68cf6 Ready kubernetes2 csi-isilon.dellemc.com rg-e0e401e1-54dc-4983-9353-183d19b68cf6 true FAILEDOVER [root@master-1-PPRvj6ITxSpjq repctl]# ./repctl --rg rg-e0e401e1-54dc-4983-9353-183d19b68cf6 reprotect --at kubernetes2 [2022-12-07 10:21:58] INFO RG (rg-e0e401e1-54dc-4983-9353-183d19b68cf6), successfully updated with action: reprotect [2022-12-07 10:22:00] INFO Cluster: kubernetes2 +-----+ | RG | +-----+ Name State rClusterID Driver RemoteRG IsSource LinkState rg-e0e401e1-54dc-4983-9353-183d19b68cf6 REPROTECT_LOCAL_IN_PROGRESS sl-k8s csi-isilon.dellemc.com rg-e0e401e1-54dc-4983-9353-183d19b68cf6 false FAILEDOVER [2022-12-07 10:22:00] INFO [2022-12-07 10:22:00] INFO Cluster: sl-k8s +-----+ | RG | +-----+ Name State rClusterID Driver RemoteRG IsSource LinkState rg-e0e401e1-54dc-4983-9353-183d19b68cf6 Ready kubernetes2 csi-isilon.dellemc.com rg-e0e401e1-54dc-4983-9353-183d19b68cf6 true FAILEDOVER [2022-12-07 10:27:46] INFO Cluster: kubernetes2 +-----+ | RG | +-----+ Name State rClusterID Driver RemoteRG IsSource LinkState rg-e0e401e1-54dc-4983-9353-183d19b68cf6 Ready sl-k8s csi-isilon.dellemc.com rg-e0e401e1-54dc-4983-9353-183d19b68cf6 true SYNCHRONIZED [2022-12-07 10:27:46] INFO [2022-12-07 10:27:46] INFO Cluster: sl-k8s +-----+ | RG | +-----+ Name State rClusterID Driver RemoteRG IsSource LinkState rg-e0e401e1-54dc-4983-9353-183d19b68cf6 Ready kubernetes2 csi-isilon.dellemc.com rg-e0e401e1-54dc-4983-9353-183d19b68cf6 false SYNCHRONIZED --------On a different K8s clusters, performing unplanned failover by blocking source array IP from k8s clusters:------ [root@master-1-xzmwDvByBkVAf ~]# echo "pre-block IP" pre-block IP [root@master-1-xzmwDvByBkVAf ~]# /root/git/csm-replication/repctl/repctl get rg [2022-12-05 14:08:41] INFO listing replication groups [2022-12-05 14:08:42] INFO Cluster: ccoff-k8s +-----+ | RG | +-----+ Name State rClusterID Driver RemoteRG IsSource LinkState rg-ecb86ad6-eca1-44ba-a0c9-29a96f5d63e3 Ready rep-k8s-2 csi-isilon.dellemc.com rg-ecb86ad6-eca1-44ba-a0c9-29a96f5d63e3 true SYNCHRONIZED [2022-12-05 14:08:42] INFO [2022-12-05 14:08:42] INFO Cluster: rep-k8s-2 +-----+ | RG | +-----+ Name State rClusterID Driver RemoteRG IsSource LinkState rg-ecb86ad6-eca1-44ba-a0c9-29a96f5d63e3 Ready ccoff-k8s csi-isilon.dellemc.com rg-ecb86ad6-eca1-44ba-a0c9-29a96f5d63e3 false SYNCHRONIZED [2022-12-05 14:08:42] INFO [root@master-1-xzmwDvByBkVAf ~]# echo "post-block IP, pre-FO" post-block IP, pre-FO [root@master-1-xzmwDvByBkVAf ~]# /root/git/csm-replication/repctl/repctl get rg [2022-12-05 14:10:56] INFO listing replication groups [2022-12-05 14:10:56] INFO Cluster: ccoff-k8s +-----+ | RG | +-----+ Name State rClusterID Driver RemoteRG IsSource LinkState rg-ecb86ad6-eca1-44ba-a0c9-29a96f5d63e3 Ready rep-k8s-2 csi-isilon.dellemc.com rg-ecb86ad6-eca1-44ba-a0c9-29a96f5d63e3 true UNKNOWN [2022-12-05 14:10:56] INFO [2022-12-05 14:10:56] INFO Cluster: rep-k8s-2 +-----+ | RG | +-----+ Name State rClusterID Driver RemoteRG IsSource LinkState rg-ecb86ad6-eca1-44ba-a0c9-29a96f5d63e3 Ready ccoff-k8s csi-isilon.dellemc.com rg-ecb86ad6-eca1-44ba-a0c9-29a96f5d63e3 false UNKNOWN [2022-12-05 14:10:56] INFO [root@master-1-xzmwDvByBkVAf ~]# [root@master-1-xzmwDvByBkVAf ~]# [root@master-1-xzmwDvByBkVAf ~]# [root@master-1-xzmwDvByBkVAf ~]# echo "post-block IP, post-FO unplanned" post-block IP, post-FO unplanned [root@master-1-xzmwDvByBkVAf ~]# /root/git/csm-replication/repctl/repctl get rg [2022-12-05 14:14:08] INFO listing replication groups [2022-12-05 14:14:08] INFO Cluster: ccoff-k8s +-----+ | RG | +-----+ Name State rClusterID Driver RemoteRG IsSource LinkState rg-ecb86ad6-eca1-44ba-a0c9-29a96f5d63e3 Ready rep-k8s-2 csi-isilon.dellemc.com rg-ecb86ad6-eca1-44ba-a0c9-29a96f5d63e3 true UNKNOWN [2022-12-05 14:14:08] INFO [2022-12-05 14:14:08] INFO Cluster: rep-k8s-2 +-----+ | RG | +-----+ Name State rClusterID Driver RemoteRG IsSource LinkState rg-ecb86ad6-eca1-44ba-a0c9-29a96f5d63e3 Ready ccoff-k8s csi-isilon.dellemc.com rg-ecb86ad6-eca1-44ba-a0c9-29a96f5d63e3 false FAILEDOVER [2022-12-05 14:14:08] INFO [root@master-1-xzmwDvByBkVAf ~]# [root@master-1-xzmwDvByBkVAf ~]# [root@master-1-xzmwDvByBkVAf ~]# [root@master-1-xzmwDvByBkVAf ~]# echo "post-FO, unblock IP" post-FO, unblock IP [root@master-1-xzmwDvByBkVAf ~]# /root/git/csm-replication/repctl/repctl get rg [2022-12-05 14:18:43] INFO listing replication groups [2022-12-05 14:18:43] INFO Cluster: ccoff-k8s +-----+ | RG | +-----+ Name State rClusterID Driver RemoteRG IsSource LinkState rg-ecb86ad6-eca1-44ba-a0c9-29a96f5d63e3 Ready rep-k8s-2 csi-isilon.dellemc.com rg-ecb86ad6-eca1-44ba-a0c9-29a96f5d63e3 true FAILEDOVER [2022-12-05 14:18:43] INFO [2022-12-05 14:18:43] INFO Cluster: rep-k8s-2 +-----+ | RG | +-----+ Name State rClusterID Driver RemoteRG IsSource LinkState rg-ecb86ad6-eca1-44ba-a0c9-29a96f5d63e3 Ready ccoff-k8s csi-isilon.dellemc.com rg-ecb86ad6-eca1-44ba-a0c9-29a96f5d63e3 false FAILEDOVER [2022-12-05 14:18:43] INFO [root@master-1-xzmwDvByBkVAf ~]# [root@master-1-xzmwDvByBkVAf ~]# [root@master-1-xzmwDvByBkVAf ~]# [root@master-1-xzmwDvByBkVAf ~]# echo "post-FO, disable src policy" post-FO, disable src policy [root@master-1-xzmwDvByBkVAf ~]# /root/git/csm-replication/repctl/repctl get rg [2022-12-05 14:23:00] INFO listing replication groups [2022-12-05 14:23:00] INFO Cluster: ccoff-k8s +-----+ | RG | +-----+ Name State rClusterID Driver RemoteRG IsSource LinkState rg-ecb86ad6-eca1-44ba-a0c9-29a96f5d63e3 Ready rep-k8s-2 csi-isilon.dellemc.com rg-ecb86ad6-eca1-44ba-a0c9-29a96f5d63e3 true FAILEDOVER [2022-12-05 14:23:00] INFO [2022-12-05 14:23:00] INFO Cluster: rep-k8s-2 +-----+ | RG | +-----+ Name State rClusterID Driver RemoteRG IsSource LinkState rg-ecb86ad6-eca1-44ba-a0c9-29a96f5d63e3 Ready ccoff-k8s csi-isilon.dellemc.com rg-ecb86ad6-eca1-44ba-a0c9-29a96f5d63e3 false FAILEDOVER [2022-12-05 14:23:00] INFO [root@master-1-xzmwDvByBkVAf ~]# [root@master-1-xzmwDvByBkVAf ~]# [root@master-1-xzmwDvByBkVAf ~]# [root@master-1-xzmwDvByBkVAf ~]# echo "post-FB" post-FB [root@master-1-xzmwDvByBkVAf ~]# /root/git/csm-replication/repctl/repctl get rg [2022-12-05 14:30:21] INFO listing replication groups [2022-12-05 14:30:21] INFO Cluster: ccoff-k8s +-----+ | RG | +-----+ Name State rClusterID Driver RemoteRG IsSource LinkState rg-ecb86ad6-eca1-44ba-a0c9-29a96f5d63e3 Ready rep-k8s-2 csi-isilon.dellemc.com rg-ecb86ad6-eca1-44ba-a0c9-29a96f5d63e3 true SYNCHRONIZED [2022-12-05 14:30:21] INFO [2022-12-05 14:30:21] INFO Cluster: rep-k8s-2 +-----+ | RG | +-----+ Name State rClusterID Driver RemoteRG IsSource LinkState rg-ecb86ad6-eca1-44ba-a0c9-29a96f5d63e3 Ready ccoff-k8s csi-isilon.dellemc.com rg-ecb86ad6-eca1-44ba-a0c9-29a96f5d63e3 false SYNCHRONIZED