Optimize read iops caused by delta compact

[flowbehappy]

Enhancement

Currently, delta compact can cause a lot of disk read ops. Because readBlockForMinorCompaction use PageReader::read(const std::vector<PageStorage::PageReadFields> & page_fields) to read all columns data of the cftiny. And this read api does not combine the continuously reads together (another optimization opportunity).

And write amplification is also very high, especially the write workload is distributed and the data volume is large.

[JaySon-Huang]

In the following codes we sort the read entries by the offset inside file, however, the read entries could from different BlobFiles. We should sort the read entries by file_id and offset to get better read locality

tiflash/dbms/src/Storages/Page/V3/BlobStore.cpp

Lines 553 to 566 in 966e7e2

	PageMap BlobStore::read(FieldReadInfos & to_read, const ReadLimiterPtr & read_limiter)
	{
	if (to_read.empty())
	{
	return {};
	}
	ProfileEvents::increment(ProfileEvents::PSMReadPages, to_read.size());
	// Sort in ascending order by offset in file.
	std::sort(
	to_read.begin(),
	to_read.end(),
	[](const FieldReadInfo & a, const FieldReadInfo & b) { return a.entry.offset < b.entry.offset; });

tiflash/dbms/src/Storages/Page/V3/BlobStore.cpp

Lines 655 to 667 in 966e7e2

	PageMap BlobStore::read(PageIDAndEntriesV3 & entries, const ReadLimiterPtr & read_limiter)
	{
	if (entries.empty())
	{
	return {};
	}
	ProfileEvents::increment(ProfileEvents::PSMReadPages, entries.size());
	// Sort in ascending order by offset in file.
	std::sort(entries.begin(), entries.end(), [](const auto & a, const auto & b) {
	return a.second.offset < b.second.offset;
	});