Cray-HPE · mharding-hpe · Nov 18, 2024 · Nov 18, 2024
@@ -751,50 +751,78 @@ Information on how to perform backups of individual services or the entire syste
 these backups.
 
 - [System Recovery](System_Recovery/System_Recovery.md)
-- etcd
-    - [Create a Manual Backup of a Healthy Bare-Metal etcd Cluster](kubernetes/Create_a_Manual_Backup_of_a_Healthy_Bare-Metal_etcd_Cluster.md)
-    - [Create a Manual Backup of a Healthy etcd Cluster](kubernetes/Create_a_Manual_Backup_of_a_Healthy_etcd_Cluster.md)
-    - [Restore an etcd Cluster from a Backup](kubernetes/Restore_an_etcd_Cluster_from_a_Backup.md)
-    - [Repopulate Data in etcd Clusters When Rebuilding Them](kubernetes/Repopulate_Data_in_etcd_Clusters_When_Rebuilding_Them.md)
-    - [Restore Bare-Metal etcd Clusters from an S3 Snapshot](kubernetes/Restore_Bare-Metal_etcd_Clusters_from_an_S3_Snapshot.md)
-- Postgres
-    - [Restore Postgres](kubernetes/Restore_Postgres.md)
-    - [Disaster Recovery for Postgres](kubernetes/Disaster_Recovery_Postgres.md)
-- Nexus
-    - [Nexus Export and Restore](package_repository_management/Nexus_Export_and_Restore.md)
-    - [Restore Nexus Data After Data Corruption](utility_storage/Restore_Corrupt_Nexus.md)
-    - [Nexus Service Recovery](package_repository_management/Nexus_Service_Recovery.md)
-- Keycloak
-    - [Create a Backup of the Keycloak Postgres Database](security_and_authentication/Create_a_Backup_of_the_Keycloak_Postgres_Database.md)
-    - [Keycloak Service Recovery](security_and_authentication/Keycloak_Service_Recovery.md)
-- Vault
-    - [Backup and Restore Vault Clusters](security_and_authentication/Backup_and_Restore_Vault_Clusters.md)
-    - [Vault Service Recovery](security_and_authentication/Vault_Service_Recovery.md)
-- SLS
-    - [Create a Backup of the SLS Postgres Database](system_layout_service/Create_a_Backup_of_the_SLS_Postgres_Database.md)
-    - [Restore SLS Postgres Database from Backup](system_layout_service/Restore_SLS_Postgres_Database_from_Backup.md)
-    - [Restore SLS Postgres without an Existing Backup](system_layout_service/Restore_SLS_Postgres_without_an_Existing_Backup.md)
-- HSM
-    - [Create a Backup of the HSM Postgres Database](hardware_state_manager/Create_a_Backup_of_the_HSM_Postgres_Database.md)
-    - [Restore HSM Postgres from a Backup](hardware_state_manager/Restore_HSM_Postgres_from_Backup.md)
-    - [Restore HSM Postgres without a Backup](hardware_state_manager/Restore_HSM_Postgres_without_a_Backup.md)
-- Spire
-    - [Create a Backup of the Spire Postgres Database](spire/Create_a_backup_of_the_Spire_Postgres_Database.md)
-    - [Restore Spire Postgres without a Backup](spire/Restore_Spire_Postgres_without_a_Backup.md)
-    - [Spire Service Recovery](spire/Spire_Service_Recovery.md)
-- Version Control Service (VCS)
-    - [Backup and restore data](configuration_management/Version_Control_Service_VCS.md#backup-and-restore-data)
-- Boot Orchestration Service (BOS)
-    - [Exporting and Importing BOS Data](boot_orchestration/Exporting_and_Importing_BOS_Data.md)
-- Boot Script Service (BSS)
-    - [Exporting and Importing BSS Data](boot_orchestration/Exporting_and_Importing_BSS_Data.md)
-- Configuration Management Service (CFS)
-    - [Exporting and Importing CFS Data](configuration_management/Exporting_and_Importing_CFS_Data.md)
-- Image Management Service (IMS)
-    - [Exporting and Importing IMS Data](image_management/Exporting_and_Importing_IMS_Data.md)
-- Workload managers
-    - [PBS Service Recovery](System_Recovery/PBS_Service_Recovery.md)
-    - [Slurm Service Recovery](System_Recovery/Slurm_Service_Recovery.md)
+
+### Backup and recovery: etcd
+
+- [Create a Manual Backup of a Healthy Bare-Metal etcd Cluster](kubernetes/Create_a_Manual_Backup_of_a_Healthy_Bare-Metal_etcd_Cluster.md)
+- [Create a Manual Backup of a Healthy etcd Cluster](kubernetes/Create_a_Manual_Backup_of_a_Healthy_etcd_Cluster.md)
+- [Restore an etcd Cluster from a Backup](kubernetes/Restore_an_etcd_Cluster_from_a_Backup.md)
+- [Repopulate Data in etcd Clusters When Rebuilding Them](kubernetes/Repopulate_Data_in_etcd_Clusters_When_Rebuilding_Them.md)
+- [Restore Bare-Metal etcd Clusters from an S3 Snapshot](kubernetes/Restore_Bare-Metal_etcd_Clusters_from_an_S3_Snapshot.md)
+
+### Backup and recovery: Postgres
+
+- [Restore Postgres](kubernetes/Restore_Postgres.md)
+- [Disaster Recovery for Postgres](kubernetes/Disaster_Recovery_Postgres.md)
+
+### Backup and recovery: Nexus
+
+- [Nexus Export and Restore](package_repository_management/Nexus_Export_and_Restore.md)
+- [Restore Nexus Data After Data Corruption](utility_storage/Restore_Corrupt_Nexus.md)
+- [Nexus Service Recovery](package_repository_management/Nexus_Service_Recovery.md)
+
+### Backup and recovery: Keycloak
+
+- [Create a Backup of the Keycloak Postgres Database](security_and_authentication/Create_a_Backup_of_the_Keycloak_Postgres_Database.md)
+- [Keycloak Service Recovery](security_and_authentication/Keycloak_Service_Recovery.md)
+
+### Backup and recovery: Vault
+
+- [Backup and Restore Vault Clusters](security_and_authentication/Backup_and_Restore_Vault_Clusters.md)
+- [Vault Service Recovery](security_and_authentication/Vault_Service_Recovery.md)
+
+### Backup and recovery: SLS
+
+- [Create a Backup of the SLS Postgres Database](system_layout_service/Create_a_Backup_of_the_SLS_Postgres_Database.md)
+- [Restore SLS Postgres Database from Backup](system_layout_service/Restore_SLS_Postgres_Database_from_Backup.md)
+- [Restore SLS Postgres without an Existing Backup](system_layout_service/Restore_SLS_Postgres_without_an_Existing_Backup.md)
+
+### Backup and recovery: HSM
+
+- [Create a Backup of the HSM Postgres Database](hardware_state_manager/Create_a_Backup_of_the_HSM_Postgres_Database.md)
+- [Restore HSM Postgres from a Backup](hardware_state_manager/Restore_HSM_Postgres_from_Backup.md)
+- [Restore HSM Postgres without a Backup](hardware_state_manager/Restore_HSM_Postgres_without_a_Backup.md)
+
+### Backup and recovery: Spire
+
+- [Create a Backup of the Spire Postgres Database](spire/Create_a_backup_of_the_Spire_Postgres_Database.md)
+- [Restore Spire Postgres without a Backup](spire/Restore_Spire_Postgres_without_a_Backup.md)
+- [Spire Service Recovery](spire/Spire_Service_Recovery.md)
+
+### Backup and recovery: Version Control Service (VCS)
+
+- [Backup and restore data](configuration_management/Version_Control_Service_VCS.md#backup-and-restore-data)
+
+### Backup and recovery: Boot Orchestration Service (BOS)
+
+- [Exporting and Importing BOS Data](boot_orchestration/Exporting_and_Importing_BOS_Data.md)
+
+### Backup and recovery: Boot Script Service (BSS)
+
+- [Exporting and Importing BSS Data](boot_orchestration/Exporting_and_Importing_BSS_Data.md)
+
+### Backup and recovery: Configuration Management Service (CFS)
+
+- [Exporting and Importing CFS Data](configuration_management/Exporting_and_Importing_CFS_Data.md)
+
+### Backup and recovery: Image Management Service (IMS)
+
+- [Exporting and Importing IMS Data](image_management/Exporting_and_Importing_IMS_Data.md)
+
+### Backup and recovery: Workload managers
+
+- [PBS Service Recovery](System_Recovery/PBS_Service_Recovery.md)
+- [Slurm Service Recovery](System_Recovery/Slurm_Service_Recovery.md)
 
 ## Multi-tenancy
 

@@ -7,32 +7,43 @@
 ## Prerequisites
 
 - Ensure that the `cray` command line interface (CLI) is authenticated and configured to talk to system management services.
-  - See [Configure the Cray CLI](../configure_cray_cli.md).
+    - See [Configure the Cray CLI](../configure_cray_cli.md).
 - The latest CSM documentation RPM must be installed on the node where the procedure is being performed.
-  - See [Check for latest documentation](../../update_product_stream/README.md#check-for-latest-documentation).
+    - See [Check for latest documentation](../../update_product_stream/README.md#check-for-latest-documentation).
 - If importing both CFS and VCS data, the VCS import should be done before the CFS import.
 
 ## Export
 
-1. (`ncn-mw#`) Run the following script to create a backup of the current CFS components, configurations, options, and sessions.
+1. (`ncn-mw#`) Run the following script to create a backup of the current CFS components, configurations, options, sessions, and sources.
 
    ```bash
    /usr/share/doc/csm/scripts/operations/configuration/export_cfs_data.sh
    ```
 
-   Expected output resembles the following:
+   Expected output ends in a line resembling the following:
 
    ```text
-   Exporting CFS components...
-   Exporting CFS configurations...
-   Exporting CFS options...
-   Exporting CFS sessions...
-   Creating compressed archive of exported data...
    SUCCESS: CFS data stored in file: /tmp/cfs-export-20230410170613-Tg0nap.tgz
    ```
 
 1. Copy the archive file it outputs to a safe location.
 
+1. Back up Vault data, if any CFS sources are defined.
+
+   CFS sources store their credentials in Vault. If any CFS sources are defined, then a CFS export is not complete without also
+   backing up the Vault data.
+
+   1. (`ncn-mw#`) Check if any CFS sources are defined.
+
+      ```bash
+      cray cfs v3 sources list --limit 1
+      ```
+
+   1. If any sources are defined, backup the Vault data.
+
+      For information on Vault backup and restore, see
+      [Backup and recovery: Vault](../README.md#backup-and-recovery-vault).
+
 ## Import
 
 An automated tool is provided to import data from the archive file created by the `export_cfs_data.sh` script.
@@ -42,17 +53,22 @@ An automated tool is provided to import data from the archive file created by th
 This tool does the following things:
 
 - The live system will have its CFS options modified to match those in the archive file, for any that differ.
+- For all CFS sources in the archive file, if a CFS source with the same name does not exist on the live system, then the
+  CFS source will be restored on the live system.
+    - Note: The restore does not recreate the associated Vault secret, which contains the source credentials.
+      Backup and restore of the Vault data must be handled separately. For information on Vault backup and restore, see
+      [Backup and recovery: Vault](../README.md#backup-and-recovery-vault).
 - For all CFS configurations in the archive file, if a CFS configuration with the same name does not exist on the live
   system, then the CFS configuration will be created on the live system.
-  - Note: No validation is performed of the layers of these configurations to ensure that their repository links and commit hashes
-    exist on the system.
+    - Note: No validation is performed of the layers of these configurations to ensure that their repository links and commit hashes
+      exist on the system.
 - For all CFS components in the archive file, their desired configurations will be updated onto the corresponding component on the live
   system if all of the following criteria are met:
-  - The component exists in CFS on the live system.
-  - The CFS component in the archive has a desired configuration set.
-  - The CFS component on the live system does NOT have a desired configuration set.
-  - The desired configuration for this component in the archive either already exists on the live system, or is going to be created
-    as part of this import process.
+    - The component exists in CFS on the live system.
+    - The CFS component in the archive has a desired configuration set.
+    - The CFS component on the live system does NOT have a desired configuration set.
+    - The desired configuration for this component in the archive either already exists on the live system, or is going to be created
+      as part of this import process.
 - If the `--clear-cfs` option is specified, then before deciding which changes need to be imported, the tool will delete all
   configurations in CFS and will clear the state, desired configuration, and error counts of all components in CFS.
 
@@ -62,6 +78,11 @@ criteria listed above.
 
 ### Import tool procedure
 
+1. Restore Vault data, if necessary.
+
+   For information on Vault backup and restore, see
+   [Backup and recovery: Vault](../README.md#backup-and-recovery-vault).
+
 1. Copy the file generated by the export script on the node where the import procedure is being performed.
 
 1. (`ncn-mw#`) Run the following script to import the data from the archive file.