backend: dynamically calculate the maximum auto-inc ID

[kennytm]

What problem does this PR solve?

Fix #222 (= TOOL-1472).

What is changed and how it works?

Remove the piece of code that pre-calculates the final AUTO_INCREMENT ID. Instead, let the encoder determines the actual value dynamically. This gives a tighter upper bound of the final AUTO_INCREMENT value and avoids overflowing AUTO_INCREMENT columns of shorter integer size (this doesn't work if PRIMARY KEY is not a handle column though, since _tidb_rowid will share the same AUTO_INCREMENT sequence)

Check List

Tests

• Integration test

Side effects

Related changes

[kennytm]

/run-all-tests

[lance6716]

This RebaseAutoID is called at every row in my understanding, why could this
rebase-per-row gives tighter upper bound than rebase-chunk-max, I think they will eventually reach almost same auto-id(rebase4Unsigned, rebase4Signed, Rebase) and now it's more expensive.

[kennytm]

The change allowing a tighter bound is deleting

tidb-lightning/lightning/restore/restore.go

Lines 649 to 653 in 4eb74ec

	for _, engine := range cp.Engines {
	for _, chunk := range engine.Chunks {
	cp.AllocBase = mathutil.MaxInt64(cp.AllocBase, chunk.Chunk.RowIDMax)
	}
	}

since RowIDMax is normally much larger than the actual row number.

Yes this could become more expensive, but we should perform a measurement to ensure. I bet the CAS loop is still much cheaper than the actual KV encoding.

[lance6716]

Now i see rebased value is cast from rowId, which is smaller or equal than row's content. And I agree CAS is lightweight. I don't know if we could move this rebase-per-row outside, such as on chunk's EOF, so there's fewer repeat.

tidb-lightning/lightning/restore/restore.go

Lines 1660 to 1661 in ce5fa5f

	case io.EOF:
	break outside

[lance6716]

CAS is lightweight so maybe we shouldn't bother moving it outside ⬆️

[kennytm]

PTAL @lance6716 @WangXiangUSTC

[lance6716]

LGTM

[IANTHEREAL]

LGTM

[IANTHEREAL]

/run-all-tests