lightning: split large csv file if possible

[XuHuaiyu]

What problem does this PR solve?

Split a large csv file if possible.

What is changed and how it works?

Split a large csv file into multiple regions, each region is about 256MB.
Then the multiple regions can be processed concurrently, and the import phase should be accelerated.

Before this commit, we need to take 40m15s to restore a 33G a tpcc csv file.
After this commit, we only need to take 8m15s to restore the same file.

Check List

Tests

• Unit test

• Manual test (add detailed scripts or steps below)
  case 1:

1. import data using lightning with this feature
   select count(*) from t; --> 60,000,000

2. import data using lightning without this feature
   select count(*) from tbl_name; --> 60,000,000

case 2:

mysql> select c_since from bmsql_customer where c_w_id = 18 and c_d_id=1 and c_id=1;
+---------------------+
| c_since             |
+---------------------+
| 2020-02-26 20:06:00 | -- select the first line of csv file, result is correct
+---------------------+
1 row in set (2.26 sec)

mysql> select c_since from bmsql_customer where c_w_id = 2000 and c_d_id=10 and c_id=3000;
+---------------------+
| c_since             |
+---------------------+
| 2020-02-26 20:42:29 | -- select the last line of csv file, result is correct
+---------------------+

Side effects

N/A

Related changes

N/A

[kennytm]

Rest LGTM

PTAL @3pointer

[siddontang]

have we used some checksum mechanisms to make esure the correctness of all data?

[3pointer]

Suggested change

	dataFileSizes = make([]float64, 0, dataFileSize/maxRegionSize)
	dataFileSizes = make([]float64, 0, dataFileSize/maxRegionSize+1)

[CLAassistant]

All committers have signed the CLA.

[XuHuaiyu]

PTAL @3pointer

[3pointer]

LGTM

[july2993]

for record:

[2020/03/13 00:15:19.981 +08:00] [INFO] [restore.go:492] [progress] [files="1280/40 (3200.0%)"] [tables="0/10 (0.0%)"] [speed(MiB/s)=117.93346221716635] [state=post-processing] []

note the progress files="1280/40 (3200.0%) is wrong, guess it's caused by split files but having not to check it.

[XuHuaiyu]

@july2993 https://github.com/pingcap/tidb-lightning/pull/272/files#diff-9697521ce51bfb4c9ef44a387e73c780R339-R353