cherry pick #2946 to release-4.0 (#2994)

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

Co-authored-by: kennytm <kennytm@gmail.com>

tidb-lightning/deploy-tidb-lightning.md

@@ -19,7 +19,7 @@ Before starting TiDB Lightning, note that:
 - If `tidb-lightning` crashes, the cluster is left in "import mode". Forgetting to switch back to "normal mode" can lead to a high amount of uncompacted data on the TiKV cluster, and cause abnormally high CPU usage and stall. You can manually switch the cluster back to "normal mode" via the `tidb-lightning-ctl` tool:
 
     ```sh
-    bin/tidb-lightning-ctl -switch-mode=normal
+    bin/tidb-lightning-ctl --switch-mode=normal
     ```
 
 - TiDB Lightning is required to have the following privileges in the downstream TiDB:
@@ -201,18 +201,13 @@ Refer to the [TiDB enterprise tools download page](/download-ecosystem-tools.md#
     # Log level: trace, debug, info, warn, error, off.
     log-level = "info"
 
+    # Listening address of the status server.
+    status-server-address = "0.0.0.0:8286"
+
     [server]
     # The listening address of tikv-importer. tidb-lightning needs to connect to
     # this address to write data.
-    addr = "192.168.20.10:8287"
-
-    [metric]
-    # The Prometheus client push job name.
-    job = "tikv-importer"
-    # The Prometheus client push interval.
-    interval = "15s"
-    # The Prometheus Pushgateway address.
-    address = ""
+    addr = "0.0.0.0:8287"
 
     [import]
     # The directory to store engine files.

tidb-lightning/migrate-from-csv-using-tidb-lightning.md

@@ -16,6 +16,7 @@ A CSV file representing a whole table must be named as `db_name.table_name.csv`.
 restored as a table `table_name` inside the database `db_name`.
 
 If a table spans multiple CSV files, they should be named like `db_name.table_name.003.csv`.
+The number part do not need to be continuous, but must be increasing and zero-padded.
 
 The file extension must be `*.csv`, even if the content is not separated by commas.
 
@@ -25,7 +26,8 @@ CSV files are schema-less. To import them into TiDB, a table schema must be prov
 done either by:
 
 * Providing a file named `db_name.table_name-schema.sql` containing the `CREATE TABLE` DDL
-    statement
+    statement, and also a file named `db_name-schema-create.sql` containing the `CREATE DATABASE`
+    DDL statement.
 * Creating the empty tables directly in TiDB in the first place, and then setting
     `[mydumper] no-schema = true` in `tidb-lightning.toml`.
 
@@ -150,6 +152,33 @@ TiDB Lightning does not support every option supported by the `LOAD DATA` statem
 * The header cannot be simply skipped (`IGNORE n LINES`), it must be valid column names if present.
 * Delimiters and separators can only be a single ASCII character.
 
+## Strict format
+
+Lightning works the best when the input files have uniform size around 256 MB. When the input is a
+single huge CSV file, Lightning can only use one thread to process it, which slows down import speed
+a lot.
+
+This can be fixed by splitting the CSV into multiple files first. For the generic CSV format, there
+is no way to quickly identify when a row starts and ends without reading the whole file. Therefore,
+Lightning by default does *not* automatically split a CSV file. However, if you are certain that the
+CSV input adheres to certain restrictions, you can enable the `strict-format` setting to allow
+Lightning to split the file into multiple 256 MB-sized chunks for parallel processing.
+
+```toml
+[mydumper]
+strict-format = true
+```
+
+Currently, a strict CSV file means every field occupies only a single line. In the other words, one
+of the following must be true:
+
+* Delimiter is empty, or
+* Every field does not contain CR (`\r`) or LF (`\n`).
+
+If a CSV file is not strict, but `strict-format` was wrongly set to `true`, a field spanning
+multiple lines may be cut in half into two chunks, causing parse failure, or even worse, quietly
+importing corrupted data.
+
 ## Common configurations
 
 ### CSV

tidb-lightning/monitor-tidb-lightning.md

@@ -14,27 +14,7 @@ Both `tidb-lightning` and `tikv-importer` supports metrics collection via [Prome
 - If TiDB Lightning is installed using TiDB Ansible, simply add the servers to the `[monitored_servers]` section in the `inventory.ini` file. Then the Prometheus server can collect their metrics.
 - If TiDB Lightning is manually installed, follow the instructions below.
 
-### `tikv-importer`
-
-`tikv-importer` v2.1 uses [Pushgateway](https://github.com/prometheus/pushgateway) to deliver
-metrics. Configure `tikv-importer.toml` to recognize the Pushgateway with the following settings:
-
-```toml
-[metric]
-
-# The Prometheus client push job name.
-job = "tikv-importer"
-
-# The Prometheus client push interval.
-interval = "15s"
-
-# The Prometheus Pushgateway address.
-address = ""
-```
-
-### `tidb-lightning`
-
-The metrics of `tidb-lightning` can be gathered directly by Prometheus as long as it is discovered. You can set the metrics port in `tidb-lightning.toml`:
+The metrics of `tidb-lightning` and `tikv-importer` can be gathered directly by Prometheus as long as it is discovered. You can set the metrics port in `tidb-lightning.toml`:
 
 ```toml
 [lightning]
@@ -44,14 +24,24 @@ pprof-port = 8289
 ...
 ```
 
-You need to configure Prometheus to make it discover the `tidb-lightning` server. For instance, you can directly add the server address to the `scrape_configs` section:
+and in `tikv-importer.toml`:
+
+```toml
+# Listening address of the status server.
+status-server-address = '0.0.0.0:8286'
+```
+
+You need to configure Prometheus to make it discover the servers. For instance, you can directly add the server address to the `scrape_configs` section:
 
 ```yaml
 ...
 scrape_configs:
-  - job_name: 'tidb-lightning'
+  - job_name: 'lightning'
     static_configs:
       - targets: ['192.168.20.10:8289']
+  - job_name: 'tikv-importer'
+    static_configs:
+      - targets: ['192.168.20.9:8286']
 ```
 
 ## Grafana dashboard

tidb-lightning/tidb-lightning-configuration.md

@@ -64,6 +64,15 @@ table-concurrency = 6
 # medium, this value might need to be adjusted for optimal performance.
 io-concurrency = 5
 
+[security]
+# Specifies certificates and keys for TLS connections within the cluster.
+# Public certificate of the CA. Leave empty to disable TLS.
+# ca-path = "/path/to/ca.pem"
+# Public certificate of this service.
+# cert-path = "/path/to/lightning.pem"
+# Private key of this service.
+# key-path = "/path/to/lightning.key"
+
 [checkpoint]
 # Whether to enable checkpoints.
 # While importing data, TiDB Lightning records which tables have been imported, so
@@ -141,6 +150,20 @@ no-schema = false
 # schema encoding.
 character-set = "auto"
 
+# Assumes the input data are "strict" to speed up processing.
+# Implications of strict-format = true are:
+#  * in CSV, every value cannot contain literal new lines (U+000A and U+000D, or \r and \n) even
+#    when quoted, i.e. new lines are strictly used to separate rows.
+# Strict format allows Lightning to quickly locate split positions of a large file for parallel
+# processing. However, if the input data is not strict, it may split a valid data in half and
+# corrupt the result.
+# The default value is false for safety over speed.
+strict-format = false
+
+# If strict-format is true, Lightning will split large CSV files into multiple chunks to process in
+# parallel. max-region-size is the maximum size of each chunk after splitting.
+# max-region-size = 268_435_456 # Byte (default = 256 MB)
+
 # Configures how CSV files are parsed.
 [mydumper.csv]
 # Separator between fields, should be an ASCII character.
@@ -184,11 +207,29 @@ index-serial-scan-concurrency = 20
 checksum-table-concurrency = 16
 
 # The default SQL mode used to parse and execute the SQL statements.
-sql-mode = "STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION"
+sql-mode = "ONLY_FULL_GROUP_BY,NO_ENGINE_SUBSTITUTION"
 # Sets maximum packet size allowed for SQL connections.
 # Set this to 0 to automatically fetch the `max_allowed_packet` variable from server on every connection.
 max-allowed-packet = 67_108_864
 
+# Whether to use TLS for SQL connections. Valid values are:
+#  * ""            - force TLS (same as "cluster") if [tidb.security] section is populated, otherwise same as "false"
+#  * "false"       - disable TLS
+#  * "cluster"     - force TLS and verify the server's certificate with the CA specified in the [tidb.security] section
+#  * "skip-verify" - force TLS but do not verify the server's certificate (insecure!)
+#  * "preferred"   - same as "skip-verify", but if the server does not support TLS, fallback to unencrypted connection
+# tls = ""
+
+# Specifies certificates and keys for TLS-enabled MySQL connections.
+# Defaults to a copy of the [security] section.
+# [tidb.security]
+# Public certificate of the CA. Set to empty string to disable TLS for SQL.
+# ca-path = "/path/to/ca.pem"
+# Public certificate of this service. Default to copy of `security.cert-path`
+# cert-path = "/path/to/lightning.pem"
+# Private key of this service. Default to copy of `security.key-path`
+# key-path = "/path/to/lightning.key"
+
 # When data importing is complete, tidb-lightning can automatically perform
 # the Checksum, Compact and Analyze operations. It is recommended to leave
 # these as true in the production environment.
@@ -231,14 +272,19 @@ log-file = "tikv-importer.log"
 # Log level: trace, debug, info, warn, error, off.
 log-level = "info"
 
+# Listening address of the status server. Prometheus can scrape metrics from this address.
+status-server-address = "0.0.0.0:8286"
+
 [server]
 # The listening address of tikv-importer. tidb-lightning needs to connect to
 # this address to write data.
-addr = "192.168.20.10:8287"
+addr = "0.0.0.0:8287"
 # Size of the thread pool for the gRPC server.
 grpc-concurrency = 16
 
 [metric]
+# These settings are relevant when using Prometheus Pushgateway. Normally you should let Prometheus
+# to scrape metrics from the status-server-address.
 # The Prometheus client push job name.
 job = "tikv-importer"
 # The Prometheus client push interval.
@@ -266,6 +312,12 @@ compression-per-level = ["lz4", "no", "no", "no", "no", "no", "lz4"]
 # (same as above)
 compression-per-level = ["lz4", "no", "no", "no", "no", "no", "lz4"]
 
+[security]
+# The path for TLS certificates. Empty string means disabling secure connections.
+# ca-path = ""
+# cert-path = ""
+# key-path = ""
+
 [import]
 # The directory to store engine files.
 import-dir = "/mnt/ssd/data.import/"
@@ -300,7 +352,7 @@ min-available-ratio = 0.05
 | -d *directory* | Directory of the data dump to read from | `mydumper.data-source-dir` |
 | -L *level* | Log level: debug, info, warn, error, fatal (default = info) | `lightning.log-level` |
 | --backend *backend* | [Delivery backend](/tidb-lightning/tidb-lightning-tidb-backend.md) (`importer` or `tidb`) | `tikv-importer.backend` |
-| --log-file *file* | Log file path | `lightning.log-file` |
+| --log-file *file* | Log file path (default = a temporary file in `/tmp`) | `lightning.log-file` |
 | --status-addr *ip:port* | Listening address of the TiDB Lightning server | `lightning.status-port` |
 | --importer *host:port* | Address of TiKV Importer | `tikv-importer.addr` |
 | --pd-urls *host:port* | PD endpoint address | `tidb.pd-addr` |
@@ -309,6 +361,15 @@ min-available-ratio = 0.05
 | --tidb-status *port* | TiDB status port (default = 10080) | `tidb.status-port` |
 | --tidb-user *user* | User name to connect to TiDB | `tidb.user` |
 | --tidb-password *password* | Password to connect to TiDB | `tidb.password` |
+| --no-schema | Ignore schema files, get schema directly from TiDB | `mydumper.no-schema` |
+| --enable-checkpoint *bool* | Whether to enable checkpoints (default = true) | `checkpoint.enable` |
+| --analyze *bool* | Analyze tables after importing (default = true) | `post-restore.analyze` |
+| --checksum *bool* | Compare checksum after importing (default = true) | `post-restore.checksum` |
+| --check-requirements *bool* | Check cluster version compatibility before starting (default = true) | `lightning.check-requirements` |
+| --ca *file* | CA certificate path for TLS connection | `security.ca-path` |
+| --cert *file* | Certificate path for TLS connection | `security.cert-path` |
+| --key *file* | Private key path for TLS connection | `security.key-path` |
+| --server-mode | Start Lightning in server mode | `lightning.server-mode` |
 
 If a command line parameter and the corresponding setting in the configuration file are both provided, the command line parameter will be used. For example, running `./tidb-lightning -L debug --config cfg.toml` would always set the log level to "debug" regardless of the content of `cfg.toml`.
 
@@ -320,6 +381,7 @@ This tool can execute various actions given one of the following parameters:
 |:----|:----|
 | --compact | Performs a full compaction |
 | --switch-mode *mode* | Switches every TiKV store to the given mode: normal, import |
+| --fetch-mode | Prints the current mode of every TiKV store |
 | --import-engine *uuid* | Imports the closed engine file from TiKV Importer into the TiKV cluster |
 | --cleanup-engine *uuid* | Deletes the engine file from TiKV Importer |
 | --checkpoint-dump *folder* | Dumps current checkpoint as CSVs into the folder |
@@ -338,6 +400,7 @@ Additionally, all parameters of `tidb-lightning` described in the section above
 | -C, --config *file* | Reads configuration from *file*. If not specified, the default configuration would be used. | |
 | -V, --version | Prints program version | |
 | -A, --addr *ip:port* | Listening address of the TiKV Importer server | `server.addr` |
+| --status-server *ip:port* | Listening address of the status server | `status-server-address` |
 | --import-dir *dir* | Stores engine files in this directory | `import.import-dir` |
 | --log-level *level* | Log level: trace, debug, info, warn, error, off | `log-level` |
 | --log-file *file* | Log file path | `log-file` |

tidb-lightning/tidb-lightning-faq.md

@@ -135,7 +135,15 @@ It is not recommended to directly use `nohup` in the command line to start `tidb
 
 ## Why my TiDB cluster is using lots of CPU resources and running very slowly after using TiDB Lightning?
 
-If `tidb-lightning` abnormally exited, the cluster might be stuck in the "import mode", which is not suitable for production. You can force the cluster back to "normal mode" using the following command:
+If `tidb-lightning` abnormally exited, the cluster might be stuck in the "import mode", which is not suitable for production. The current mode can be retrieved using the following command:
+
+{{< copyable "shell-regular" >}}
+
+```sh
+tidb-lightning-ctl --fetch-mode
+```
+
+You can force the cluster back to "normal mode" using the following command:
 
 {{< copyable "shell-regular" >}}
 

tidb-lightning/tidb-lightning-glossary.md

@@ -63,7 +63,9 @@ See also the [Troubleshooting guide](/troubleshoot-tidb-lightning.md#checksum-fa
 
 ### Chunk
 
-Equivalent to a single file in the data source.
+A continuous range of source data, normally equivalent to a single file in the data source.
+
+When a file is too large, Lightning may split a file into multiple chunks.
 
 ### Compaction
 

troubleshoot-tidb-lightning.md

@@ -27,7 +27,18 @@ There are several reasons why TiDB Lightning becomes slow:
 
 Every additional index will introduce a new KV pair for each row. If there are N indices, the actual size to be imported would be approximately (N+1) times the size of the Mydumper output. If the indices are negligible, you may first remove them from the schema, and add them back via `CREATE INDEX` after import is complete.
 
-**Cause 3**: TiDB Lightning is too old.
+**Cause 3**: Each file is too large.
+
+TiDB Lightning works the best when the data source are broken down into multiple files of size around 256 MB, so that the data can be processed in parallel. Lightning seems unresponsive if each file is too large.
+
+If the data source is CSV, and all CSV files have no fields containing literal new line characters (U+000A and U+000D), you can turn on "strict format" to let Lightning automatically split the large files.
+
+```toml
+[mydumper]
+strict-format = true
+```
+
+**Cause 4**: TiDB Lightning is too old.
 
 Try the latest version! Maybe there is new speed improvement.