tools: update docs for Lightning (#843)

* tools/lightning: the region-min-size config has been removed

* tools/lightning: included the IO concurrency settings and related metrics

* tools/lightning: removed mention of mixed deployment

* tools/lightning: added TiKV free space requirement

Added several FAQ entries explaining the tight requirements

* tools/lightning: removed one more sentence related to mixed deployment

* tools/lightning: stop using /tmp to store the engine files

* Apply suggestions from code review

Co-Authored-By: kennytm <kennytm@gmail.com>

tools/lightning/deployment.md

@@ -21,18 +21,15 @@ Before starting TiDB-Lightning, note that:
 
 ## Hardware requirements
 
-`tidb-lightning` and `tikv-importer` are both resource-intensive programs. It is recommended to deploy them into two separate machines. If your hardware resources are limited, you can deploy `tidb-lightning` and `tikv-importer` on the same machine.
-
-### Hardware requirements of separate deployment
+`tidb-lightning` and `tikv-importer` are both resource-intensive programs. It is recommended to deploy them into two separate machines.
 
 To achieve the best performance, it is recommended to use the following hardware configuration:
 
 - `tidb-lightning`:
 
     - 32+ logical cores CPU
-    - 16 GB+ memory
-    - 1 TB+ SSD, preferring higher read speed
-    - 10 Gigabit network card
+    - An SSD large enough to store the entire SQL dump, preferring higher read speed
+    - 10 Gigabit network card (capable of transferring at ≥300 MB/s)
     - `tidb-lightning` fully consumes all CPU cores when running,
         and deploying on a dedicated machine is highly recommended.
         If not possible, `tidb-lightning` could be deployed together with other components like
@@ -42,28 +39,19 @@ To achieve the best performance, it is recommended to use the following hardware
 
     - 32+ logical cores CPU
     - 32 GB+ memory
-    - 1 TB+ SSD, preferring higher IOPS
-    - 10 Gigabit network card
+    - 1 TB+ SSD, preferring higher IOPS (≥ 8000 is recommended)
+    - 10 Gigabit network card (capable of transferring at ≥300 MB/s)
     - `tikv-importer` fully consumes all CPU, disk I/O and network bandwidth when running,
         and deploying on a dedicated machine is strongly recommended.
-        If not possible, `tikv-importer` could be deployed together with other components like
-        `tikv-server`, but the import speed might be affected.
 
 If you have sufficient machines, you can deploy multiple Lightning/Importer servers, with each working on a distinct set of tables, to import the data in parallel.
 
-### Hardware requirements of mixed deployment
-
-If your hardware resources are severely under constraint, it is possible to deploy `tidb-lightning` and `tikv-importer` and other components on the same machine, but the import performance is affected.
-
-It is recommended to use the following configuration of the single machine:
-
-- 32+ logical cores CPU
-- 32 GB+ memory
-- 1 TB+ SSD, preferring higher IOPS
-- 10 Gigabit network card
-
 > **Notes:** `tidb-lightning` is a CPU intensive program. In an environment with mixed components, the resources allocated to `tidb-lightning` must be limited. Otherwise, other components might not be able to run. It is recommended to set the `region-concurrency` to 75% of CPU logical cores. For instance, if the CPU has 32 logical cores, you can set the `region-concurrency` to 24.
 
+Additionally, the target TiKV cluster should have enough space to absorb the new data.
+Besides [the standard requirements](../../op-guide/recommendation.md), the total free space of the target TiKV cluster should be larger than **Size of SQL dump × [Number of replicas](../../FAQ.md#is-the-number-of-replicas-in-each-region-configurable-if-yes-how-to-configure-it) × 2**.
+With the default replica count of 3, this means the total free space should be at least 6 times the size of SQL dump.
+
 ## Export data
 
 Use the [`mydumper` tool](../../tools/mydumper.md) to export data from MySQL by using the following command:
@@ -225,7 +213,7 @@ Download the TiDB-Lightning package (choose the same version as that of the TiDB
 
     [import]
     # The directory to store engine files.
-    import-dir = "/tmp/tikv/import"
+    import-dir = "/mnt/ssd/data.import/"
     # Number of threads to handle RPC requests.
     num-threads = 16
     # Number of concurrent import jobs.
@@ -268,11 +256,18 @@ Download the TiDB-Lightning package (choose the same version as that of the TiDB
     # This value affects the memory usage of tikv-importer.
     # Must not exceed the max-open-engines setting for tikv-importer.
     table-concurrency = 8
+
     # The concurrency number of data. It is set to the number of logical CPU
     # cores by default. When deploying together with other components, you can
     # set it to 75% of the size of logical CPU cores to limit the CPU usage.
     #region-concurrency =
 
+    # The maximum I/O concurrency. Excessive I/O concurrency causes an increase in
+    # I/O latency because the disk's internal buffer is frequently refreshed,
+    # which causes the cache miss and slows down the read speed. Depending on the storage
+    # medium, this value might need to be adjusted for optimal performance.
+    io-concurrency = 5
+
     # Logging
     level = "info"
     file = "tidb-lightning.log"
@@ -313,9 +308,6 @@ Download the TiDB-Lightning package (choose the same version as that of the TiDB
     # Block size for file reading. Keep it longer than the longest string of
     # the data source.
     read-block-size = 65536 # Byte (default = 64 KB)
-    # Each data file is split into multiple chunks of this size. Each chunk
-    # is processed in parallel.
-    region-min-size = 268435456 # Byte (default = 256 MB)
     # mydumper local source data directory
     data-source-dir = "/data/my_database"
     # If no-schema is set to true, tidb-lightning assumes that the table skeletons

tools/lightning/errors.md

@@ -12,7 +12,7 @@ This document summarizes some commonly encountered errors in the `tidb-lightning
 
 ## Import speed is too slow
 
-Normally it takes Lightning 2 minutes per thread to import a 256 MB Chunk. It is an error if the speed is much slower than this. The time taken for each chunk can be checked from the log mentioning `restore chunk … takes`. This can also be observed from metrics on Grafana.
+Normally it takes Lightning 2 minutes per thread to import a 256 MB data file. It is an error if the speed is much slower than this. The time taken for each data file can be checked from the log mentioning `restore chunk … takes`. This can also be observed from metrics on Grafana.
 
 There are several reasons why Lightning becomes slow:
 

tools/lightning/faq.md

@@ -104,4 +104,17 @@ If `tidb-lightning` abnormally exited, the cluster might be stuck in the "import
 
 ```sh
 tidb-lightning-ctl --switch-mode=normal
-```
+```
+
+## Can TiDB-Lightning be used with 1-Gigabit network card?
+
+The TiDB-Lightning toolset requires a 10-Gigabit network card. The 1-Gigabit network card is *not acceptable*, especially for `tikv-importer`.
+A 1-Gigabit network card can only provide a total bandwidth of 120 MB/s, which has to be shared among all target TiKV stores.
+TiDB-Lightning can easily saturate all bandwidth of the 1-Gigabit network, and brings down the cluster because PD is unable to contact it anymore.
+
+## Why TiDB-Lightning requires so much free space in the target TiKV cluster?
+
+With the default settings of 3 replicas, the space requirement of the target TiKV cluster is 6 times the size of SQL dump. The extra multiple of “2” is a conservative estimation because the following factors are not reflected in the SQL dump:
+
+- The space occupied by indices
+- Space amplification in RocksDB

tools/lightning/monitor.md

@@ -96,9 +96,9 @@ Metrics provided by `tidb-lightning` are listed under the namespace `lightning_*
 
 - **`lightning_idle_workers`** (Gauge)
 
-    Counting idle workers. Values should be less than the `table-concurrency`/`region-concurrency` settings and are typically zero. Labels:
+    Counting idle workers. Values should be less than the `*-concurrency` settings and are typically zero. Labels:
 
-    - **name**: `table` / `region`
+    - **name**: `table` / `region` / `io`
 
 - **`lightning_kv_encoder`** (Counter)
 
@@ -143,6 +143,20 @@ Metrics provided by `tidb-lightning` are listed under the namespace `lightning_*
 
     Bucketed histogram of the size of a block of KV pairs.
 
+- **`lightning_chunk_parser_read_block_seconds`** (Histogram)
+
+    Bucketed histogram of the time needed by the data file parser to read a block.
+
+- **`lightning_chunk_parser_read_row_seconds`** (Histogram)
+
+    Bucketed histogram of the time needed by the data file parser to read a row.
+
 - **`lightning_checksum_seconds`** (Histogram)
 
     Bucketed histogram of the time taken to compute the checksum of a table.
+
+- **`lightning_apply_worker_seconds`** (Histogram)
+
+    Bucketed histogram of the time taken to acquire an idle worker. Labels:
+
+    - **name**: `table` / `region` / `io`