lightning: sample files and pre-allocate rowID before restoring chunk

[buchuitoudegou]

What problem does this PR solve?

Issue Number: close #28776

Problem Summary:

Currently, Lightning pre-allocates rowIDs for each row when populating chunks according to data file sizes. But it occurs that estimates of rowID are now conservatively too large so that Lightning usually pre-allocates a large number of rowIDs, which significantly exceeds what we need. This issue may result in:

1. if the number of rows is relatively close to the maximum of the data type of its auto_increment field, Lightning is likely to allocate rowIDs that exceed this limit. (e.g. 2^31 rows of data may probably have 2^32 rowIDs, which exceeds the range of int32 if the auto_increment field is int32)
2. Too much space of rowID is wasted because of over-estimate.

What is changed and how it works?

In this pr, I'm about to:

1. sample rows from data files and do a better estimation.
2. re-allocate ids when they are run out (due to bad estimation)
3. early stop when the rowID overflows

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• D3Hunter
• gozssky

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[buchuitoudegou]

/component lightning

[CLAassistant]

All committers have signed the CLA.

[sre-bot]

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/92f2e521694dc63c6ca141cb5a41203396bff86f

[buchuitoudegou]

/cc @gozssky @D3Hunter @Ehco1996

PTAL

[buchuitoudegou]

/run-integration-br-test

[buchuitoudegou]

ptal @gozssky

[sleepymole]

Suggested change

	case mysql.TypeLonglong:
	if !isUnsigned {
	return rowID > math.MaxInt64
	}
	// impossible for rowID exceeding MaxUint64
	return uint64(rowID) > math.MaxUint64

[sleepymole]

/hold

[buchuitoudegou]

/run-integration-br-test

[sleepymole]

/unhold
/merge

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: 5a839c9

[ti-srebot]

cherry pick to release-4.0 in PR #35456

[ti-srebot]

cherry pick to release-6.0 in PR #35457

[ti-srebot]

cherry pick to release-6.1 in PR #35458