Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merge master beyond 8.4(f520bdfbc32ea) #398

Merged
merged 146 commits into from
Nov 7, 2024
Merged

Merge master beyond 8.4(f520bdfbc32ea) #398

merged 146 commits into from
Nov 7, 2024

Conversation

CalvinNeo
Copy link
Member

What is changed and how it works?

Issue Number: Close #xxx

What's Changed:

Related changes

  • PR to update pingcap/docs/pingcap/docs-cn:
  • Need to cherry-pick to the release branch

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Release note

hbisheng and others added 30 commits July 17, 2024 01:23
@hbisheng
Revert "raftstore: handle region destroy in a separate thread pool (t…
097aa4f 
…ikv#17121)" (tikv#17275)

close tikv#17258

This reverts commit 273ec86.

The previous commit introduced a bug that could result in data loss. 
Specifically, the region worker might delete data from a range where a 
snapshot has just been applied.

Signed-off-by: Bisheng Huang <hbisheng@gmail.com>
@LykxSassinator
metrics: fix incorrect unit of grpc source requests' duration. (tikv#…
254ceaa 
…17279)

ref tikv#16265

Fix incorrect format unit of the `grpc source requests' duration` panel.

Signed-off-by: lucasliang <nkcs_lykx@hotmail.com>
@LykxSassinator
Revert "metrics: fix incorrect unit of grpc source requests' duration(t…
ac37609 
…ikv#17279)" (tikv#17280)

ref tikv#16265

Revert duplicated changes.

Signed-off-by: lucasliang <nkcs_lykx@hotmail.com>
@ti-chi-bot
doc: make c++ complier requirement more accurate (tikv#17266)
cc2f344 
close tikv#17265

Signed-off-by: Wenqi Mou <wenqimou@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@overvenus
*: ignore unsound openssl::bio::MemBio::get_buf (tikv#17292)
60c3271 
close tikv#17291

Ignore RUSTSEC-2024-0357 (unsound `openssl::bio::MemBio::get_buf`) as we
have banned relevant APIs in the TiKV codebase.

Also, TiKV temporarily pins OpenSSL 1.1.1 because OpenSSL 3.0.0 has
performance regressions. See details in deny.toml.

Signed-off-by: Neil Shen <overvenus@gmail.com>
@SpadeA-Tang
In-memory engine: fix gc may filter keys of onging snapshot (tikv#17288)
7146b71 
close tikv#17282

fix gc may filter keys of onging snapshot

Signed-off-by: SpadeA-Tang <tangchenjie1210@gmail.com>
@SpadeA-Tang @ti-chi-bot
In-memory engine: fix repeatable schedule delete-range for the same r…
d60c7fb 
…ange (tikv#17261)

close tikv#17259

fix repeatable schedule delete-range for the same range

Signed-off-by: SpadeA-Tang <tangchenjie1210@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@purelind @ti-chi-bot
chore: bump nightly version to 8.3.0-alpha (tikv#17289)
dc76d90 
ref tikv#16890

Signed-off-by: purelind <purelind@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@Connor1996
raftstore: Fix too large slow log (tikv#17295)
5b3bdf7 
close tikv#17294

Fix too large slow log

Signed-off-by: Connor1996 <zbk602423539@gmail.com>
@LykxSassinator @ti-chi-bot
raftstore: optimize the batching mechanism when enabling async-ios. (t…
ccd84b4 
…ikv#17029)

ref tikv#16907

Introduce a simple algorithm to make the batching mechanism on WriteBatch larger as expected,
for reducing the cost on IO resources.

Signed-off-by: lucasliang <nkcs_lykx@hotmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@SpadeA-Tang
In-memory engine: add eviction duration metrics (tikv#17321)
6488a12 
ref tikv#16141

add eviction duration metrics

Signed-off-by: SpadeA-Tang <tangchenjie1210@gmail.com>
@cfzjywxk
txn: refactor scheduler and add more information for error type (tikv…
b700fba 
…#17297)

ref tikv#14474

1. Refactor the `ProcessWrite` function in the transaction scheduler.
2. Add `start_ts` information to the `AlreadyExist` error types, so it would be easier
to know the conflict write record and related transactions.
3. Abstract memory quota count into task allocation, ensure the re-scheduled
commands and tasks are considered by the memory quota statistics.

Signed-off-by: cfzjywxk <cfzjywxk@gmail.com>
@hicqu @ti-chi-bot
cdc: re-design delegate and downstream to support partial region subs…
b834dac 
…cription better (tikv#17274)

close tikv#16362

Signed-off-by: qupeng <qupeng@pingcap.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@mittalrishabh @ti-chi-bot
TiKV : Stale reads.: add histogram of min safe ts. (tikv#17308)
92da45a 
close tikv#17307

Signed-off-by: rishabh_mittal <mittalrishabh@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@Tristan1900 @ti-chi-bot
br: use dashmap for br router#tasks (tikv#17301)
2253c93 
close tikv#17300

Signed-off-by: Wenqi Mou <wenqimou@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@ti-chi-bot @wuhuizuo
OWNERS: Auto Sync OWNERS files from community membership (tikv#17343)
e0199a0 
 

Signed-off-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

Co-authored-by: wuhuizuo <wuhuizuo@126.com>
@Connor1996
batch-system: Reduce the memory usage of peers' message channel (tikv…
7c50908 
…#17326)

close tikv#16229

Reduce the memory usage of peers' message channel

Signed-off-by: Connor1996 <zbk602423539@gmail.com>
@SpadeA-Tang @ti-chi-bot
In-memory engine: show index/select_by_range_cache_engine in handle/w…
f1ff5e2 
…ait duration in cop detail (tikv#17324)

ref tikv#16141

show index/select-by-range-cache-engine in handle/wait duration in cop detail

Signed-off-by: SpadeA-Tang <tangchenjie1210@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@LykxSassinator @ti-chi-bot @hbisheng
*: enlarge the default region-size. (tikv#17311)
74a760f 
ref tikv#17309

This pr make the default setting of `coprocessor.region-split-size` larger,
from `96MB` to `256MB` by default.  The new setting for `region-split-size` is
compatible to the larger cluster.

Signed-off-by: lucasliang <nkcs_lykx@hotmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Co-authored-by: Bisheng Huang <hbisheng@gmail.com>
@SpadeA-Tang
In-memory engine: show eviction reasons in metrics (tikv#17345)
0a7fb7c 
ref tikv#16141

show eviction reasons in metrics

Signed-off-by: SpadeA-Tang <tangchenjie1210@gmail.com>
@SpadeA-Tang @ti-chi-bot
In-memory engine: gc the range immediately after load (tikv#17298)
78d53dc 
close tikv#17299

gc the range immediately after load

Signed-off-by: SpadeA-Tang <tangchenjie1210@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@v01dstar
RocksDB: Fix bloom filter incompatible issue (tikv#17361)
050f1a6 
close tikv#17272

TiKV no longer names bloom filter blocks with suffix like "FullBloom" or "Ribbon".

Signed-off-by: Yang Zhang <yang.zhang@pingcap.com>
@LykxSassinator @ti-chi-bot
*: fix compatibility to raft-engine.spill-dir (tikv#17358)
0e38069 
close tikv#17356

Make the diskfull check mechanism compatible to the configuration `raft-engine.spill-dir`.

Signed-off-by: lucasliang <nkcs_lykx@hotmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@cfzjywxk
raftstore: debug print write info details (tikv#17348)
819eefe 
ref tikv#14474

Signed-off-by: cfzjywxk <cfzjywxk@gmail.com>
@Leavrth @ti-chi-bot
raftstore: remove the behavior - immediately persist when the command…
26c20c7 
…'s priority is different from that of apply poller. (tikv#17338)

close tikv#17278

remove the behavior - immediately persist when the command's priority is different from that of apply poller.

Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@Tristan1900
br: some code cleanup in router (tikv#17347)
918a9bc 
close tikv#17360

remove scheduler.schedule(Task::MarkFailover(Instant::now()))?; as not needed anymore
rename load_task to register_stream_task so can be more explicit of what the method is doing
rename Service to BackupStreamGrpcService to be more explicit 
rename StreamTaskInfo to StreamTaskHandler as it contains methods acting as reactors on stream task events. It's more like a handler rather than read-only info.
remove unused metrics tikv_backup_request_duration_seconds
remove unused code handle_fatal_error

Signed-off-by: Wenqi Mou <wenqimou@gmail.com>
@Connor1996
server: Fix io flow stat for snapshot sending (tikv#17384)
0f03af5 
ref tikv#15990

Fix io flow stat for snapshot sending. Its io flow is not taken into account previously 
because the io flow is recorded only when using `file_system::File` which is a wrapper of `std::fs::File`.

Signed-off-by: Connor1996 <zbk602423539@gmail.com>
@glorv @ti-chi-bot
In-Memory-Engine: move from range-base to region-base (tikv#17331)
2577ecf 
ref tikv#16141, close tikv#17147

Move range cache engine's range manager for range-base to region-base

Signed-off-by: glorv <glorvs@163.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@3AceShowHand @overvenus
cdc: print log to indicate memory free, and adjust finish_scan_lock m…
9734214 
…ethod (tikv#17357)

close tikv#17368

* add one log to indicate the memory quota is freed when drop the `Drain`
* free the truncated scanned event memory quota.
* refactor `finish_scan_lock` method, to remove the else branch.
* row size calculation should also consider old value
* remove some outdate todo

Signed-off-by: 3AceShowHand <jinl1037@hotmail.com>
Signed-off-by: Ling Jin <7138436+3AceShowHand@users.noreply.github.com>

Co-authored-by: Neil Shen <overvenus@gmail.com>
@breezewish @EricZequan @ti-chi-bot
copr: Support Vector Data Type and Functions (tikv#17303)
4f3f951 
ref tikv#17290

Signed-off-by: EricZequan <110292382+EricZequan@users.noreply.github.com>
Signed-off-by: “EricZequan” <zequany33@gmail.com>

Co-authored-by: EricZequan <110292382+EricZequan@users.noreply.github.com>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
SpadeA-Tang and others added 9 commits October 28, 2024 02:47
@SpadeA-Tang
In-memory engine: copy read throughput and latency panels to in-memor…
b0e2f87 
…y engine (tikv#17702)

ref tikv#16141

copy read throughput and latency panels to in-memory engine

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>
@SpadeA-Tang
In-memory engine: evict region when destroy peer (tikv#17712)
2798c83 
ref tikv#16141, close tikv#17644

Evict region when destroy peer

Signed-off-by: SpadeA-Tang <tangchenjie1210@gmail.com>
Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>
@SpadeA-Tang @ti-chi-bot
*: fix compile fail for test (tikv#17706)
a18c8d3 
close tikv#17707

fix compile fail for test

Signed-off-by: SpadeA-Tang <tangchenjie1210@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@v01dstar
RocksDB: Fix SST ingestion (tikv#17726)
8888085 
close tikv#17691

Update RocksDB to include fixes for ingestion

* tikv/rocksdb@31eb76b
* tikv/rocksdb@cf9cd5b

Signed-off-by: Yang Zhang <yang.zhang@pingcap.com>
@SpadeA-Tang
compaction-filter: add test for triggering compaction in CheckAndComp…
43101c5 
…act (tikv#17725)

close tikv#17269

add test for triggering compaction in CheckAndCompact

Signed-off-by: SpadeA-Tang <tangchenjie1210@gmail.com>
@overvenus @ti-chi-bot
In-memory Engine: Remove boilerplate code (tikv#17380)
31a2160 
ref tikv#17181

This commit is a follow-up of tikv#17359. Since `HybridEngine` is now
integrated into TiKV through coprocessor observers and RaftKv, it no
longer needs to implement the Engine traits. This commit removes the
unnecessary boilerplate code for `HybridEngine`.

Signed-off-by: Neil Shen <overvenus@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@purelind
chore: bump master version to 8.5.0-alpha (tikv#17729)
78e7966 
ref tikv#16890

Signed-off-by: purelind <purelind@gmail.com>
@v01dstar
RocksDB: Fix deprecated option (tikv#17732)
845a71b 
close tikv#17731

Add deprecated RocksDB option to avoid panic upgrading from TiKV 6.1.x to 8.4+

Signed-off-by: Yang Zhang <yang.zhang@pingcap.com>
@CalvinNeo
Merge remote-tracking branch 'tikv/master' into merge-master-beyond-8.4
549464e
@CalvinNeo CalvinNeo changed the title Merge master beyond 8.4(f520bdfbc32ea17035552e3476dedeaf612061c1) Merge master beyond 8.4(f520bdfbc32ea) Oct 31, 2024
@CalvinNeo
Copy link
Member Author

/pull-unit-test

@wuhuizuo
Copy link

wuhuizuo commented Nov 5, 2024

/retest

4 similar comments
@CalvinNeo
Copy link
Member Author

/retest

@CalvinNeo
Copy link
Member Author

/retest

@CalvinNeo
Copy link
Member Author

/retest

@purelind
Copy link

purelind commented Nov 5, 2024

/retest

@purelind
Copy link

purelind commented Nov 5, 2024

/test pull-unit-test

@CalvinNeo CalvinNeo force-pushed the merge-master-beyond-8.4 branch 3 times, most recently from a8d98da to 549464e Compare November 5, 2024 14:02
@purelind
Copy link

purelind commented Nov 6, 2024

/retest

2 similar comments
@purelind
Copy link

purelind commented Nov 6, 2024

/retest

@purelind
Copy link

purelind commented Nov 7, 2024

/retest

@ti-chi-bot ti-chi-bot bot added the lgtm label Nov 7, 2024
@ti-chi-bot ti-chi-bot
Copy link

ti-chi-bot bot commented Nov 7, 2024

[LGTM Timeline notifier]

Timeline:

  • 2024-11-07 07:37:27.333025367 +0000 UTC m=+1112960.172180913: ☑️ agreed by Lloyd-Pottiger.

@ti-chi-bot ti-chi-bot bot added the approved label Nov 7, 2024
@CalvinNeo
Copy link
Member Author

/approve

@ti-chi-bot ti-chi-bot
Copy link

ti-chi-bot bot commented Nov 7, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: CalvinNeo, Lloyd-Pottiger

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@CalvinNeo CalvinNeo merged commit b877a97 into pingcap:raftstore-proxy Nov 7, 2024
4 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Lloyd-Pottiger Lloyd-Pottiger Lloyd-Pottiger approved these changes

Assignees
No one assigned
Labels
approved lgtm size/XXL
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.