Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add backup&restore #1100

Merged
merged 8 commits into from
Feb 16, 2022
7 changes: 2 additions & 5 deletions docs-2.0/7.data-security/2.backup-restore/2.compile-br.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ To compile the BR, do a check of these:

To compile the BR, follow these steps:

1. Clone the `nebula-storage` repository to your machine.
1. Clone the `nebula-br` repository to your machine.

```bash
git clone https://github.com/vesoft-inc/nebula-br.git
Expand All @@ -35,8 +35,5 @@ Users can enter `bin/br version` on the command line. If the following results a

```bash
[nebula-br]$ bin/br version
Nebula Backup And Restore Ultility Tool,V-0.1.0
GitSha: 8e994dc
GitRef: master
please run "help" subcommand for more infomation.
Nebula Backup And Restore Utility Tool,V-0.6.0
```
136 changes: 87 additions & 49 deletions docs-2.0/7.data-security/2.backup-restore/4.br-restore-data.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,67 +16,105 @@ To restore data with the BR, do a check of these:

- The BR is compiled. For more information, see [Compile BR](2.compile-br.md).

- No application is connected to the target Nebula Graph cluster.

- Make sure that the target and the source Nebula Graph clusters have the same topology, which means that they have exactly the same number of hosts.

- Your account on the BR machine can log on to the meta and the storage servers via SSH without a password. For more information, see[SSH tunnels with keys](http://alexander.holbreich.org/ssh-tunnel-without-password/).

!!! caution

This account must have write privileges to the installation directory of Nebula Graph and the data directory (--data_path) of the meta/storage service.

- To perform the restoration process, a `root` user is required.
- Download [nebula-agent](https://github.com/vesoft-inc/nebula-agent) and start the agent service in each cluster(including metad, storaged, graphd) host.

- If the backup files are stored on Alibaba Cloud OSS or Amazon S3, make sure that the S3 CLI client or ossutil is installed and configured on the meta servers, the storage servers, and the BR machine. For more information, see [Alibaba Cloud ossutil Documentation](https://www.alibabacloud.com/help/zh/doc-detail/120075.htm#concept-303829) and [Amazon S3 CLI Documentation](https://docs.amazonaws.cn/cli/latest/userguide/cli-services-s3.html).

!!! caution
- No application is connected to the target Nebula Graph cluster.

Run `ln -s /<ossutil_tool_installation_path>/<ossutil64 or ossutil> /usr/local/bin/ossutil` to make the `ossutil` command effective.
- Make sure that the target and the source Nebula Graph clusters have the same topology, which means that they have exactly the same number of hosts. The number of data folders for each host is consistently distributed.
foesa-yang marked this conversation as resolved.
Show resolved Hide resolved

## Procedures

1. Users can use the following command to list the existing backup information:

```bash
$ ./bin/br show --storage <ip_address>
```
For example, use the following command to list the backup information in the local `/home/test/backup` path.
```bash
$ ./bin/br show --storage "local:///home/test/backup"
+----------------------------+---------------------+------------------+-------------+---------------+
| NAME | CREATE TIME | SPACES | FULL BACKUP | SPECIFY SPACE |
+----------------------------+---------------------+------------------+-------------+---------------+
| BACKUP_2021_07_29_06_51_09 | 2021-07-29 06:51:09 | basketballplayer | true | true |
| BACKUP_2021_07_29_07_55_08 | 2021-07-29 07:55:09 | basketballplayer | true | true |
+----------------------------+---------------------+------------------+-------------+---------------+
```
```bash
$ ./bin/br show --storage <ip_address>
```
For example, use the following command to list the backup information in the local `/home/nebula/backup` path.
foesa-yang marked this conversation as resolved.
Show resolved Hide resolved
```bash
$ ./bin/br show --storage "local:///home/nebula/backup"
+----------------------------+---------------------+------------------------+-------------+------------+
| NAME | CREATE TIME | SPACES | FULL BACKUP | ALL SPACES |
+----------------------------+---------------------+------------------------+-------------+------------+
| BACKUP_2022_02_10_07_40_41 | 2022-02-10 07:40:41 | basketballplayer | true | true |
| BACKUP_2022_02_11_08_26_43 | 2022-02-11 08:26:47 | basketballplayer,foesa | true | true |
+----------------------------+---------------------+------------------------+-------------+------------+
```

Or, you can use the following command to list the backup information in S3 URL `s3://127.0.0.1:9000/br-test/backup`.
foesa-yang marked this conversation as resolved.
Show resolved Hide resolved
```bash
$ ./bin/br show --s3.endpoint "http://127.0.0.1:9000" --storage="s3://br-test/backup/" --s3.access_key=minioadmin --s3.secret_key=minioadmin --s3.region=default
```

| Parameter | Data type | Required | Default value | Description |
foesa-yang marked this conversation as resolved.
Show resolved Hide resolved
| --- | --- | --- | --- | --- |
| `-h,-help` | - | No | None | Checks help for restoration. |
| `-debug` | - | No | None | Checks for more log information. |
| `-log` | string | No | br.log | Specify br detail log path. |
foesa-yang marked this conversation as resolved.
Show resolved Hide resolved
| `--storage` | string | Yes | None | The target storage url of BR backup data. The format is: <Schema\>://<PATH\>. <br>Schema: Optional values are `local` and `s3`. <br>When selecting s3, you need to fill in `s3.access_key`, `s3.endpoint`, `s3.region`, and `s3.secret_key`.<br>PATH: The path of the storage location. |
foesa-yang marked this conversation as resolved.
Show resolved Hide resolved
| `--s3.access_key` | string | No | None | Set access key id. |
foesa-yang marked this conversation as resolved.
Show resolved Hide resolved
| `--s3.endpoint` | string | No | None | Set the S3 endpoint URL, please specify the http or https scheme explicitly. |
foesa-yang marked this conversation as resolved.
Show resolved Hide resolved
| `--s3.region` | string | No | None | Set region or location to upload or download backup. |
foesa-yang marked this conversation as resolved.
Show resolved Hide resolved
| `--s3.secret_key` | string | No | None | Set secret key for access id. |
foesa-yang marked this conversation as resolved.
Show resolved Hide resolved


2. Run the following command to restore data.

```
[root]# ./bin/br restore full --meta <ip_address> --storage <storage_path> --name <backup_name> --user <user_name>
```
```
$ ./bin/br restore full --meta <ip_address> --storage <storage_path> --name <backup_name>
```

For example, use the following command to upload the backup files in the local `/home/nebula/backup/` to the cluster where the meta service's address is `127.0.0.1:9559`.
foesa-yang marked this conversation as resolved.
Show resolved Hide resolved

For example, use the following command to restore the data in the local `/home/foesa/backup` path to the cluster where the address is `192.168.*.*:9559` and the user name of the host where the meta/storage service is located is `root`.
```
$ ./bin/br restore full --meta "127.0.0.1:9559" --storage "local:///home/nebula/backup/" --name BACKUP_2021_12_08_18_38_08
```

```
[root]# ./bin/br restore full --meta "192.168.*.*:9559" --storage "local:///home/test/backup" --name "BACKUP_2021_07_29_07_55_08" --user "root"
```
Or, you can use the following command to upload the backup files in S3 URL `s3://127.0.0.1:9000/br-test/backup`.
foesa-yang marked this conversation as resolved.
Show resolved Hide resolved
```bash
$ ./bin/br restore full --meta "127.0.0.1:9559" --s3.endpoint "http://127.0.0.1:9000" --storage="s3://br-test/backup/" --s3.access_key=minioadmin --s3.secret_key=minioadmin --s3.region="default" --name BACKUP_2021_12_08_18_38_08
```

If the following information is returned, the data is restored successfully.
```bash
restore successed
```
If the following information is returned, the data is restored successfully.
```bash
Restore succeed.
```

The parameters are as follows.
!!! caution

| Parameter | Required | Default value | Description | Data type |
| --- | --- | --- | --- | --- |
| --concurrent | No | 5 | The maximum number of concurrency for Alibaba Cloud OSS. | int |
| --extra_args | No | None | The parameters of backup storage tools (OSS/HDFS/S3) for backup. | string |
| -h,-help | No | None | Checks help for restoration. | - |
| --meta | Yes | None | The IP address and port of the meta service. | string |
| --name | Yes | None | The backup name. | string |
| --storage | Yes | None | The target storage url of BR backup data. | string |
| --user | Yes | None | The username to login into the hosts where meta/storage service is located. | string |
If your new cluster hosts' ip are not all the same with the backup cluster, after restore, you should use `add hosts` to add the Storage hosts in the new cluster one by one.
foesa-yang marked this conversation as resolved.
Show resolved Hide resolved

The parameters are as follows.

| Parameter | Data type | Required | Default value | Description |
foesa-yang marked this conversation as resolved.
Show resolved Hide resolved
| --- | --- | --- | --- | --- |
| `-h,-help` | - | No | None | Checks help for restoration. |
| `-debug` | - | No | None | Checks for more log information. |
| `-log` | string | No | br.log | Specify br detail log path. |
| `-meta` | string | Yes | None | The IP address and port of the meta service. |
| `-name` | string | Yes | None | The name of backup. |
| `--storage` | string | Yes | None | The target storage url of BR backup data. The format is: \<Schema\>://\<PATH\>. <br>Schema: Optional values are `local` and `s3`. <br>When selecting s3, you need to fill in `s3.access_key`, `s3.endpoint`, `s3.region`, and `s3.secret_key`.<br>PATH: The path of the storage location. |
| `--s3.access_key` | string | No | None | Set access key id. |
| `--s3.endpoint` | string | No | None | Set the S3 endpoint URL, please specify the http or https scheme explicitly. |
| `--s3.region` | string | No | None | Set region or location to upload or download backup. |
| `--s3.secret_key` | string | No | None | Set secret key for access id. |

3. Run the following command to clean up temporary files if any error occurred during backup. It will clean the files in cluster and external storage. You could also use it to clean up old backups files in external storage.

```bash
$ ./bin/br cleanup --meta <ip_address> --storage <storage_path> --name <backup_name>
```

The parameters are as follows.

| Parameter | Data type | Required | Default value | Description |
| --- | --- | --- | --- | --- |
| `-h,-help` | - | No | None | Checks help for restoration. |
| `-debug` | - | No | None | Checks for more log information. |
| `-log` | string | No | br.log | Specify br detail log path. |
| `-meta` | string | Yes | None | The IP address and port of the meta service. |
| `-name` | string | Yes | None | The name of backup. |
| `--storage` | string | Yes | None | The target storage url of BR backup data. The format is: \<Schema\>://\<PATH\>. <br>Schema: Optional values are `local` and `s3`. <br>When selecting s3, you need to fill in `s3.access_key`, `s3.endpoint`, `s3.region`, and `s3.secret_key`.<br>PATH: The path of the storage location. |
| `--s3.access_key` | string | No | None | Set access key id. |
| `--s3.endpoint` | string | No | None | Set the S3 endpoint URL, please specify the http or https scheme explicitly. |
| `--s3.region` | string | No | None | Set region or location to upload or download backup. |
| `--s3.secret_key` | string | No | None | Set secret key for access id. |
11 changes: 5 additions & 6 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -334,11 +334,11 @@ nav:
- User management: 7.data-security/1.authentication/2.management-user.md
- Roles and privileges: 7.data-security/1.authentication/3.role-list.md
- OpenLDAP authentication: 7.data-security/1.authentication/4.ldap.md
# - Backup & Restore:
# - What is Backup & Restore: 7.data-security/2.backup-restore/1.what-is-br.md
# - Compile BR: 7.data-security/2.backup-restore/2.compile-br.md
# - Use BR to back up data: 7.data-security/2.backup-restore/3.br-backup-data.md
# - Use BR to restore data: 7.data-security/2.backup-restore/4.br-restore-data.md
- Backup & Restore:
- What is Backup & Restore: 7.data-security/2.backup-restore/1.what-is-br.md
- Compile BR: 7.data-security/2.backup-restore/2.compile-br.md
- Use BR to back up data: 7.data-security/2.backup-restore/3.br-backup-data.md
- Use BR to restore data: 7.data-security/2.backup-restore/4.br-restore-data.md
- Manage snapshots: 7.data-security/3.manage-snapshot.md
- SSL: 7.data-security/4.ssl.md

Expand Down Expand Up @@ -546,7 +546,6 @@ plugins:
# Exclude files with unix-style wildcards (globs)
glob:
# Exclude all files in a directory. The path starts with the directory name in docs-2.0, such as `20.appendix/*`.
- 7.data-security/2.backup-restore/*
- nebula-flink/*
- CHANGELOG.md
- spark-connector/*
Expand Down