Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SONiC configuration setup enhancements to provide installer migration… #590

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
108 changes: 83 additions & 25 deletions doc/ztp/SONiC-config-setup.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# SONiC Configuration Setup Service

# High Level Design Document
#### Rev 0.2
#### Rev 0.4

# Table of Contents
* [List of Tables](#list-of-tables)
@@ -17,6 +17,7 @@
* [2.2 Functional Description](#22-functional-description)
* [2.3 CLI](#23-cli)
* [3. Unit Tests](#3-unit-tests)
* [4. Appendix](#4-appendix)


# List of Tables
@@ -27,6 +28,8 @@
|:---:|:-----------:|:------------------:|-----------------------------------|
| 0.1 | 07/16/2019 | Rajendra Dendukuri | Initial version |
| 0.2 | 07/22/2019 | Rajendra Dendukuri | Update Test plan, fixed review comments |
| 0.3 | 10/16/2019 | Rajendra Dendukuri | Added an example usecase of password migration |
| 0.4 | 02/25/2020 | Rajendra Dendukuri | Added:<BR> Installer migration hooks capability<BR> Automatic backup and restore of migration hooks|

# About this Manual
This document provides details about how the switch configuration is handled on a SONiC device.
@@ -45,11 +48,11 @@ This document describes functional behavior of the proposed config-setup service

# 1 Feature Overview

SONiC switch configuration is stored in a redis database instance called Config DB. The contents of Config DB reflect most of the configuration of a SONiC switch. The contents of Config DB can be saved in a file */etc/sonic/config_db.json* using the *config save* CLI command. During switch bootup, Config DB is populated with the intended configuration present in the file */etc/sonic/config_db.json*. Through out this document the term startup-configuration is used to refer to */etc/sonic/config_db.json*.
SONiC switch configuration is stored in a Redis database instance called Config DB. The contents of Config DB reflect most of the configuration of a SONiC switch. The contents of Config DB can be saved in a file */etc/sonic/config_db.json* using the *config save* CLI command. During switch bootup, Config DB is populated with the intended configuration present in the file */etc/sonic/config_db.json*. Through out this document the term startup-config is used to refer to */etc/sonic/config_db.json*.



When a new SONiC firmware version is installed, the newly installed image does not include a *startup-configuration*. A startup-config has to be created on first boot. Also when the user upgrades from firmware version A to version B, the startup-config needs to be migrated to the new version B.
When a new SONiC firmware version is installed from ONIE, the newly installed image does not include a *startup-config*. A startup-config has to be created on first boot. Also when the user upgrades from firmware version A to version B, the startup-config needs to be migrated to the new version B.



@@ -79,7 +82,7 @@ A new service, ***config-setup***, is being proposed to take care of all of the

6. It must provide infrastructure to support config migration when the user installs a new SONiC firmware version and reboots to switch to the newly installed version.

7. The config-setup service must also take into consideration of other methods to configuring a switch. e.g ZTP, updategraph.
7. The config-setup service must also take into consideration of other methods of configuring the switch. e.g ZTP, updategraph.



@@ -104,7 +107,7 @@ When the switch reboots in warm boot mode, *config-setup* service must ensure th



In additional to this borrowed functionality, few provisions for user defined extensions (hooks) have been added to perform customizations which are implementation based.
In addition to this borrowed functionality, few provisions for user defined extensions (hooks) have been added to perform customizations which are implementation based.



@@ -128,11 +131,11 @@ The *config-setup* service is used to perform the following functions:

### 2.2.1 Config Setup Hooks

To extend the functionality of the *config-setup* script, users are not expected to modify this script directly. Instead users are required to place executable *hook* scripts in corresponding hooks directory which extend the config-setup script's capability. All executable scripts in the hooks directory are invoked inline using the Bourne shell '.' command. The exit status of the config-setup action will be passed to each hook in the *exit_status* shell variable, and will always be zero if the script succeeded at the task for which it was invoked. The hook scripts can modify the value of exit_status to change the exit status of the config-setup action.
To extend the functionality of the *config-setup* script, users are not expected to modify this script directly. Instead users are required to place executable *hook* scripts in corresponding hooks directory which extend the config-setup script's capability. All executable scripts in the hooks directory are invoked inline using the Bourne shell '.' command. The hook scripts are invoked in lexicographical order of their filenames. The exit status of the hook script is captured in the shell variable *exit_status*. Before each hook is executed, the shell variable *exit_status* is set to zero. If the hook script succeeded at the task for which it was invoked, the value remains as zero. The hook scripts can modify the value of exit_status to indicate the exit status if any error is encountered.



With in a hook script, if required a system reboot operation can be performed. If such an event occurs, the config-setup service will resume executing hooks from the point where reboot was issued. Any hooks previously executed will not be executed again. The satisfying condition for initiating a reboot and the reboot operation itself is not implemented in the config-setup service. Instead this logic needs to be implemented as part of the hook script.
If required, within a hook script a system reboot operation can be performed. If such an event occurs, the config-setup service will resume executing hooks from the beginning of the hook where reboot was issued. Any hooks previously executed will not be executed again. The satisfying condition for initiating a reboot and the reboot operation itself is not implemented in the config-setup service. Instead this logic needs to be implemented as part of the hook script.



@@ -148,15 +151,15 @@ At this point if ZTP is enabled, config-setup service loads the ZTP configuratio



If both ZTP and updategraph are not enabled, config-setup service is responsible to create a factory default startup-configuration for the switch and load it to Config DB.
If both ZTP and updategraph are not enabled, config-setup service is responsible to create a factory default startup-config for the switch and load it to Config DB.



#### Factory Default Config Hooks

Hooks Directory: */etc/config-setup/factory-default-hooks.d*

If defined by the user these are executed during factory default config creation step of *config-setup* service. User can choose to create the startup-config file, frr configuration files and any other files using pre-defined logic. Various application packages can install hooks with recipes to create their own factory default configuration, config-setup service will execute these recipes.
If defined by the user these are executed during factory default config creation step of *config-setup* service. User can choose to create the startup-config file, frr configuration files and any other files using pre-defined logic. Various application packages can install hooks with logic to create their own factory default configuration, config-setup service will execute these recipes.



@@ -182,27 +185,29 @@ fi

#### Preset Configuration

If a startup-config was not created as part of factory default, config-setup service uses the config preset defined in the /usr/share/sonic/devices/*$platform*/*default_sku* file. This is the current observed functionality provided by the updategraph service.
If a startup-config was not created by the factory default hooks, config-setup service uses the config preset defined in the /usr/share/sonic/device/*$platform*/*default_sku* file. This is the current observed functionality provided by the updategraph service.



### 2.2.2 Config Migration

The user can use the *sonic_installer* utility to install a new version of SONiC firmware. As part of this procedure, sonic_installer takes a backup of all files in directory */etc/sonic* and copies them as */host/old_config*. Later when switch boots into the newly installed image, a file */tmp/pending_config_migration* is created by rc.local service and config-setup service makes note of it. The backed up files in /host/old_config are then restored to /etc/sonic directory and the restored startup-config file is loaded to Config DB.
The user can use the *sonic_installer* utility to install a new version of SONiC firmware. As part of this procedure, sonic_installer takes a backup of all files in the directory */etc/sonic* and copies them as */host/old_config*. The *config-migration-pre-hooks* and the *config-migration-post-hooks.d* which are described in the subsequent sections are also backed up as part of this action.

Later when the switch boots into the newly installed image, a file */tmp/pending_config_migration* is created by rc.local service and config-setup service detects its presence and starts the configuration migration action. The backed up files in /host/old_config are then restored to /etc/sonic directory and the restored startup-config file is loaded to Config DB. The *config-migration-pre-hooks* and the *config-migration-post-hooks.d* are also restored to the newly installed image from their backed up copies. If the new image contains files in the migration hooks directories with the same filename as the backed up files, they are not overwritten and instead the files in the newly installed image will be used for configuration migration. This allows the newer version images to provide updated migration logic and deprecate the older (possibly incorrect) implementations.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If my understanding is correct, in current implementation the startup_config migration is performed by the '/usr/bin/db_migrator.py' script, what's the plan for this part of function? will the db_migrator.py be kept or have something new to replace it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The /usr/bin/db_migrator.py is responsible to migrate the contents of the startup_config. The config-setup.service, which calls /usr/bin/config-setup is responsible to migrate the startup_config file. The config-setup service calls db_migrator.py. They both work together.




#### Config migration Hooks - Pre (Take Backup)

Hooks Directory: */etc/config-setup/config-migration-pre-hooks.d*

Config migration hooks provides various applications the ability to extend their config migration step and define their own back up scripts that are invoked when */etc/sonic* is backed up. Based on the specific requirement of an application, the corresponding config-migration-pre-hook for the application implements the appropriate recipe. These hooks can be invoked by using the "config-setup backup" command.
Config migration hooks provides various applications the ability to extend their config migration step and define their own backup scripts that are invoked when */etc/sonic* is backed up by the *sonic_installer install* command. Based on the specific requirement of an application, the corresponding config-migration-pre-hook for the application implements the appropriate recipe. These hooks can also be invoked by using the "config-setup backup" command. After the *sonic_installer install* command is executed, the *config-setup backup* command is automatically invoked which in turn executes the *Config migration Hooks - Pre* scripts.



Below is an example hook to take a copy of the known ssh hosts file.

*/etc/config-setup/config-migration-pre-hooks.d/backup-known-ssh-known-hosts*
*/etc/config-setup/config-migration-pre-hooks.d/backup-known-ssh-hosts*

```bash
#Take a backup of known_hosts file for all users
@@ -214,13 +219,31 @@ for user in $(ls -1 /home); do
done
```

The *Config migration Hooks - Pre* are intended solely for the end user to add their own scripts as part of post deployment provisioning; these scripts are not to be included as part of the SONiC installer image, *sonic-xxx.bin*. Instead, it is encouraged to use the *installer migration hooks* described in the next section to be part of the SONiC installer image, *sonic-xxx.bin*.

#### Installer migration Hooks - Pre (Take Backup)

Hooks Directory: */var/run/config-setup/installer-migration-hooks*

The functionality of installer migration hooks is similar to the Config migration Hooks - Pre. These hooks are also used to perform configuration backup actions and are executed
by the *config-setup backup* command after executing the hooks defined in */etc/config-setup/config-migration-pre-hooks.d*. These hooks are packaged as part of the SONiC installer image, *sonic-xxx.bin*, and are copied to the */var/run/config-setup/* as part of the SONiC installer image extraction step. As the installer migration hooks are part of the newly installed image, they are easier to provide new configuration backup functionality and fix any migration issues exposed by the hooks defined in * Config migration Hooks - Pre*.



The installer migration hooks corresponding to an installed SONiC image are available in the directory */host/image-<build_version>/installer-migration-hooks*. If the *config-setup backup* is executed explicitly by the user and the */var/run/config-setup/installer-migration-hooks* directory is not present, the hooks present in the directory */host/image-<build_version>/installer-migration-hooks* of the current active image are executed. This allows the user to perform manual config backup and restore operation on the current active image. If the */var/run/config-setup/installer-migration-hooks* directory exists, it will take precedence as it indicates that the user has just installed a new image and is preparing to boot into the newly installed image.



#### Configuration Backup Flowchart

![](images/Config_Backup.png)


#### Config migration Hooks - Post (Restore Backup)

Hooks Directory: */etc/config-setup/config-migration-post-hooks.d*

These hooks are executed by the config-setup service when switch boots into a newly installed image and also a snapshot of backed up configuration files are found in /host/old_config directory. Applications can install corresponding hooks in the *config-migration-post-hooks.d* directory, which restore the files that were baked up in the *config-migration-pre-hooks*.
These hooks are executed by the config-setup service when switch boots into a newly installed image and also a snapshot of backed up configuration files are found in /host/old_config directory. Applications can install corresponding hooks in the *config-migration-post-hooks.d* directory, which restore the files that were backed up in the *config-migration-pre-hooks*.



@@ -243,9 +266,13 @@ rm -rf /host/ssh_config/*_known_hosts



#### Configuration Restore Flowchart

![](images/Config_Restore.png)

### 2.2.3 Config Detection

On every switch bootup, config-setup service starts and detects if a startup-config file is present or not. If startup-config does not exist, config Initialization action is performed.
On every switch bootup, the config-setup service starts and detects if a startup-config file is present or not. If startup-config does not exist, config Initialization action is performed. Please refer to the section 2.2.1 for more details.



@@ -256,7 +283,7 @@ Following are the commands supported by the config-setup tool. These are Linux s

***config-setup factory***

This command is executed to create factory default configuration. Please refer to section 2.2.1 for more details. It is to be noted that this command simply creates a configuration and does not load it into Config DB. It is up to the calling entity to execute either 'config reload' or switch reboot for the configuration to take effect.
This command is executed to create factory default configuration. Please refer to the section 2.2.1 for more details. Note that this command simply creates a configuration and does not load it into Config DB. It is up to the calling entity to execute either 'config reload' or switch reboot for the configuration to take effect.



@@ -276,16 +303,47 @@ This command is executed as part of system bootup by the config-setup service. U

1. Install SONiC firmware image on an empty disk using ONIE or similar bare metal provisioning tool. Verify that a factory default configuration is created.
2. Delete startup-config and reboot the switch. Verify that a factory default configuration is created.
3. Install a new SONiC firmware image using "sonic_installer install" command and reboot the switch to boot into newly installed image. Verify that the startup-config file used in the old SONiC firmware is restored to be used by the new image.
4. When ZTP is enabled, verify that the ZTP configuration is loaded when SONiC switch boots without startup-config. Factory default config does not get created in this case.
5. Verify that updategraph service takes over config creation role if it is enabled and SONiC switch boots without startup-config.
6. Verify that the user specified config-setup-factory hooks are executed when switch boots without a startup-config.
3. Install a new SONiC firmware image using the "sonic_installer install" command and reboot the switch to boot into the newly installed image. Verify that the startup-config file used in the old SONiC firmware is restored to be used by the new image.
4. When ZTP is enabled, verify that the ZTP configuration is loaded when the SONiC switch boots without a startup-config. Factory default config does not get created in this case.
5. Verify that the updategraph service takes over the config creation role if it is enabled and SONiC switch boots without a startup-config.
6. Verify that the user specified config-setup-factory hooks are executed when the switch boots without a startup-config.
7. Verify that the user specified config-migration-pre hooks are executed when a new SONiC firmware image is installed.
8. Verify that when the switch boots into the newly installed SONiC image, the user specified config-migration-post hooks are executed.
8. Verify that when the switch boots into a newly installed SONiC image, the user specified config-migration-post hooks are executed.
9. Verify that the exit status of the user defined hooks is correctly read and reported by the config-setup service in syslog. If the hook script is syntactically incorrect, it will be reported with failed exit status.
10. When the switch boots with a saved startup-config, verify that the config-setup service does not perform any additional actions.
11. Verify that the user can execute the "config-setup factory" command manually with requiring a switch reboot.
11. Verify that the user can execute the "config-setup factory" command manually without requiring a switch reboot.
12. Verify that the user can execute the "config-setup backup" command multiple times so that any new configuration file changes are picked up again.
13. Verify that when a config-setup hook scripts issue a switch reboot, all the hook scripts previously executed are not executed again up subsequent switch boot.
13. Verify that config-migration hook scripts are executed even when switch boots in warm boot mode. This will allow scripts to perform any actions required for some applications which are not covered by the warm boot functionality.
14. Verify that config-setup hook scripts are not executed when switch boots in warm boot mode.
13. Verify that when a config-setup hook script issues a switch reboot, all the hook scripts previously executed are not executed again upon subsequent switch boot.
14. Verify that config-migration hook scripts are executed even when the switch boots in warm boot mode. This will allow scripts to perform any actions required for some applications which are not covered by the warm boot functionality.
15. Verify that config-setup-factory hook scripts are not executed when switch boots in warm boot mode.
16. Verify that the config-migration-pre and config-migration-post hook scripts are automatically migrated to the newly installed image.
17. Verify that the config-migration-pre and config-migration-post hook scripts migrated to the newly installed image do not overwrite hook scripts with same name in the newly installed image.
18. Verify that the installer migration scripts provided as part of the SONiC installer image are executed after the executing the hooks defined in config-migration-pre. This action is performed by the *config-setup backup* command.

# 4 Appendix

This section provides example extensions of the Configuration Setup service to perform configuration migration in a SONiC switch.

## Linux Users, Groups and Passwords Migration

When a new SONIC image is installed, a new Linux root filesystem is created. As a result, any changes made in the Linux root filesystem are not carried forward to the newly installed image. Only specific contents in the /etc/sonic directory (e.g config_db.json, frr config) are migrated to the new image as part of config migration. Simple changes in Linux, such as change of admin user password or Unix group membership, are not transferred over as part of config migration.

The config-setup migration hooks infrastructure can be used to migrate following information from existing image to newly installed image:
1. Newly created non-system users (GID >= 1000)
2. Password for the newly created users and the admin user
3. Newly created user groups
4. /home directory which has all user home directories
5. User email

When a new SONiC switch image is installed using the "sonic_installer install" command, the /var/lib/config-setup/installer-migration-hooks/01-local-users-pre script extracted from the new SONiC image is executed. It takes a backup copy of all of the files listed below and copies them to /host/backup/users_info directory.

```
/etc/passwd
/etc/group
/etc/shadow
/etc/gshadow
/home/*
/var/spool/mail
```

After the SONiC switch is rebooted to boot into the newly installed image, the /etc/config-setup/config-migration-post-hooks.d/01-local-users-post script is executed which reads the files in /host/backup/users_info directory and migrates contents of the backed up data to the Linux root filesystem of newly installed user. After migrating the user data, the directory /host/backup/users_info is removed.
1 change: 1 addition & 0 deletions doc/ztp/images/Config_Backup.drawio
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
<mxfile host="" modified="2020-02-26T21:43:10.809Z" agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/11.2.5 Chrome/76.0.3809.139 Electron/6.0.7 Safari/537.36" etag="LQGhVl-CJy_cP10q9pxL" pages="1" version="11.2.5" type="device"><diagram id="C5RBs43oDa-KdzZeNtuy" name="Page-1">7Vvbcto6FP0aHun4KpvHkGtPezo9TW/pS0fYwigxFrVFAv36I9vyTTLGOJiQlk4ng2QhWXstrb23JAb6+Xx1HcLF7F/iIn+gKe5qoF8MNE01NG0Q/1fcdVpj2Xpa4YXY5Y2Kilv8G/FKhdcusYuiSkNKiE/xolrpkCBADq3UwTAkT9VmU+JXR11AD0kVtw705dpv2KWztNbWrKL+BmFvlo2sglH6ZA6zxnwm0Qy65KlUpV8O9POQEJp+mq/OkR8bL7PLt7frb/77B3D9z3/RL/hl/O7zh6/DtLOrXb6STyFEAe3cdXRp3o7W3s34LlzNvs9+gHcfroYcy0foL7m9+FzpOjMgcpk9eZGEdEY8EkD/sqgdh2QZuCgeRmGlos17QhasUmWV94jSNScHXFLCqmZ07vOnEQ3JQw4PM+x4SgKaDaq0nD43U0SWoYMa2vE5Uxh6iDbYhpsinn+JSty414jMEQ3XrEGIfEjxY5VvkNPWy9sV0LAPHJ0dSCAjFZEAOz9xEFHo+yiMO0g/SxAWAMXWfpphim4XMDHSE1v1VTAS06dIqTEUng+jiGMrA5UvCkVCjZWx758Tn4TJa+guRPbUybspPQGOjSZT9mQ3qB9RSNGqERz+VLc4i7iKqYCXnwpNULOFPivpgaH0hCeQ8Hw7HcRvcEXnC/Y3x3U4x14IMSXBcEbIQxRTcoUjyl7oSkZ6RuaTZbQdZQm3KzjHfmydG+Q/IoodWMMF6GMvYAWHwcMYV0sINiQOPFYCRelzIgRDYztHptOp5tRyxAUTYII+OQLMCkdsw5Q5otdwJCfO3kliSSS5XCFnSVEDUygumHLSAQFjoFV1wLZqdKAOY2sPENd6GRnil/DAzMzh+nv8/TdmVrzj3SWFi1WltN5IhD157lpbKbLnbvKVh/fcTW9dQviWzYDKMPs+C4pRC+3eYnRZVG0H1YvqxDYNU+lTVEfCgpPXm12z3PS+lpv6N4ORJ2UcDL3Gwx0UDa3Gv9EQsjSQDRa4cVoYIhi7OwDnseGDSbRIDAR89m7jCYt6gUeTmpM33H1xatabasyjmfL61BRTpoTZ2wKV5dIhwRR7wwjRJUNYmUDnIf5wgnNLkqMbB0xy6sGU5VZW2sA9i/d5WMmJUcCOEJqsMC1FJqx0V3pSxCVxIQtLXkc4o8nhTOOS2BrPlHA2a2DO6lqHPXyEjwQHtKCZAao0M22BPunE+bfKO1JiR0JsAFSho9QyUkcJFfNpP4OdsvvpzE61xM2CqdvYWeFmQdWjYGe2Wfu3stMUgiVTjIL6Zqe843dOFrEhuT8s4ptFGG8IZLtDLLdLdgdmJGIvdnVyly3dpaFa7dylpfflLs2XcZcHFpaaNL5xBZyUZc/K0mbPaRvNOkZYx09PbSTTs2kH/1jYqVUzOcs2O7JT2dJR3+y098nO3SKsLtHcYQ8vQUtyWsdFTrPKKVXRurJzVO1ppByYnSOJnXdI3ltiIQitUlI6OosDFexA/4w/mGPXTTfzUYR/w0keMS3iuSSzM8cD84LV+HCC/DGL67wkiqsen7F/2znYvPLEECq/ncHfalC+AFEXWilvVJBZigOVZbtdmZQ1IdNphHqBNtOZtkcxEfLmzKDlc5jWuqTupkuyxnTxo33qktUyptOMoxImIMR0I3ErrK0uWYqocGY7YWJ8getSM77aN7+xLQyk6qawGtIu97s25GS0chKN5/E1rNPu+275pypQJpPMbQkoEEi6v/MYQ4Y5PoYRVfBPPBzTbQEOVa25/6Eccvtcky8JyVi8wH5A9/Svu9dplKXX5nYMIHLNHHXzO4YusfawmwnaPjYTCo4qdiX1Gpl2I1P7OuhJh/mIQszsE4frz2Rv21yukeXHQl4dVGMmsC/u5g7wUNyVtxo+EIm9rzKXy5fls3O5IUvmhJDzeTQ6QCYnJ+lFtFp3enKKUTfeia1m8RbQ5ZiorysFYA2/3OPPd1P6K8QX3qeR9mNZc5tu46lYcv51OhbbOS3ZcDpxiGOxWshf6BJJzaZLx2ij19O0mhC4aeEcSxBhVFmW3xPdOYgQOjIyz7f/GOL+bPHVuv40Jesn9Ss+u3Hd+8lmPWK9Ieqwv8nvdGQBIr77M9WsVy9CNUFGO363FiFg5DcVDyFDtUj/iTIkqEE3XWqSm7IsbV4/L6VCuW/LaKbkNNv5YErsypC66lmJ5L27g/Bzfyn4HinZJNXb8+3RS3JSvKSvStfv21JSEw43VFW8sdKZkaxY/PA5bV78fFy//B8=</diagram></mxfile>
Binary file added doc/ztp/images/Config_Backup.png

Unable to render rich display

1 change: 1 addition & 0 deletions doc/ztp/images/Config_Restore.drawio
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
<mxfile host="" modified="2020-02-26T21:45:08.697Z" agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/11.2.5 Chrome/76.0.3809.139 Electron/6.0.7 Safari/537.36" etag="Lq45-5PJuhTdIxyEcgdM" pages="1" version="11.2.5" type="device"><diagram id="C5RBs43oDa-KdzZeNtuy" name="Page-1">7Vrbdto4FP0aHpnlq2weA4GUpk07ZaYJfekytnxpbIvIIpj5+pGx5JscYihuaVdfgnUkHayz97mJDNRJlN5ga+2/Rw4MB4rkpAP1eqAosmQC+pFJdrnEAEYu8HDgsEWlYBH8B/lOJt0EDkxqCwlCIQnWdaGN4hjapCazMEbb+jIXhfVvXVseFAQL2wpF6X3gED+XmopRyt/AwPP5N8tglM9EFl/MTpL4loO2FZE6HagTjBDJn6J0AsPMeNwu9/PdffjuEdy8/Tt5sv4d3/5z93mYK5sds6U4AoYxOVl1MtUXo533ZrzEqf/gfwG3dzO2RXq2wg2z14JYmLADkx23IlVFAaOD8dYPCFysLTub2VLOUJlPopCOZPqYEIweC0tTG43dIAwnKER4r0l1TRvadrGyMrMydU2X6EzH4zKzPENMYFoBmx3/BqIIEryjS9isarDjMiqP2HBb4QUT+RVKqExmMSZ6heLS2vSBGfwI48uqYP1PMCEIw9wf3MAbRoGHLRKgeLjOxMCKMoPHq2Rvd4Qek8wpMIoGGSdnPkooeLOVZT9u1gKKGG1iBzoMqleQdFFMmDvLGY5eaCWZBaRWlAvnkETIHQuabivkwDbhyu0Tcl2tQa7Ko790AXVZaYHd7A12IMAu+lvsXGXBj47szO6BXQcHpgF5yKxNT5OPlpWZ65QBsR/s+CCmb1/ZlA2X1bly2360exHs49BK0AbbsIMf0MjjQXJgocoMBZ1azBfRr2Crt0DLZRiG1Lee65miDW/2DR9RQA9ckEvjSYSRS5MkTi6uJD8721cNzk1VckOVLKjKrSOo2lOwOPp3sNI4AytPZNgpbO6dlcqoIysV7bJY2aASAGdjJRj9aFZy21ZYOY2d37Q8AfVcZQCxPpGlFuJofWUq7gJV66fQ3pD2AoXWHkNWkvwpPBqFBzBbwGwrO3oDUxVr/R9SdoiQnZomzhzgeTVRDfChfO1cjc0P8N1irj5oi+vPzoQ3SRca3zUxKHeP73WS6s2at+fozhHo2ADtexvpT9NzbOzRtI6x5xwtz9CY3ejap+WHISTLewXNya1bHKEaekBImMFreIGnDeITw2QPxRVdICvrtJykT172OXf3DMhVrTAX03cktE9WZmsa0ILY+5pz6WvBpcyb0yAh9Jwzvp0uz1+G62jyyEfRapOco8ZwXaW9xnDACuigT2rISo0ackuNoUiqSA1Z7osbYgw4Oi39rBRTzRyHeH8xmaMeF4oYcHzeqF+kGUZveaPVrGJTUMQA7r53qOLXqxedOkNmwYYJ9CIK+7QUjU8gHquQeE20rFVLr1zMyDWqlsw9ez10iKy/Wr8LGmQU7mu7shpIDVbr3VhNGWLtKsvW2YLk5RfWG9fQiqwffK/mesOsXfTTh/wNTnWxJzdEy4/jt5q33Ezhe3sOviktPwwILraESTcf+03vMr/LiXgovhAn0rTiXpyz0hidmBwEVZVL9/MniEMdW2tfQRVCYtO/CYopFVsaCRQ6rGIUuHzpzYRQCXaldudmQm+7leqrmWgFt62Z+AXvMRqef1oMOsT+1xP56KeGHKVBrFPztqBopJ0p2NBh+fN+vrz8Jwl1+j8=</diagram></mxfile>
Binary file added doc/ztp/images/Config_Restore.png

Unable to render rich display