Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TiFlash OOM with high throughput write #6407

Closed
flowbehappy opened this issue Dec 2, 2022 · 1 comment · Fixed by #6446
Closed

TiFlash OOM with high throughput write #6407

flowbehappy opened this issue Dec 2, 2022 · 1 comment · Fixed by #6446
Assignees
@JaySon-Huang
Labels
affects-5.0 affects-5.1 affects-5.2 affects-5.3 affects-5.4 This bug affects the 5.4.x(LTS) versions. affects-6.0 affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-6.5 This bug affects the 6.5.x(LTS) versions. component/storage severity/critical type/bug The issue is confirmed as a bug.

Comments

@flowbehappy
Copy link
Contributor

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

  1. A cluster with many TiKV and 4 TiFlash
  2. A table with shard_row_id_bits
  3. Each TiFlash instance has over 300GB
  4. The GC interval is set to 5 days
  5. Write throughput of each TiFlash is over 7MB/s

2. What did you expect to see? (Required)

TiFlash works normally.

3. What did you see instead (Required)

TiFlash OOM.
image
image

4. What is your TiFlash version? (Required)

v6.1.x
@flowbehappy flowbehappy added type/bug The issue is confirmed as a bug. affects-5.0 affects-5.1 affects-5.2 affects-5.3 affects-5.4 This bug affects the 5.4.x(LTS) versions. affects-6.0 affects-6.1 This bug affects the 6.1.x(LTS) versions. severity/critical labels Dec 2, 2022
@flowbehappy
Copy link
Contributor Author

It is very likely caused by delta compact works slowly. And the root cause is that ~Snapshot of Storage V2 is slow in the case of many page entries.
image

@flowbehappy flowbehappy mentioned this issue Dec 2, 2022
Closed
@VelocityLight VelocityLight added the affects-6.5 This bug affects the 6.5.x(LTS) versions. label Dec 2, 2022
@jebter jebter removed the affects-6.5 This bug affects the 6.5.x(LTS) versions. label Dec 2, 2022
@flowbehappy flowbehappy added component/storage affects-6.5 This bug affects the 6.5.x(LTS) versions. labels Dec 6, 2022
This was referenced Dec 8, 2022
Merged
Merged
ti-chi-bot pushed a commit that referenced this issue Dec 8, 2022
@JaySon-Huang
PageStorage: background version compact for v2 (#6446)
f248fac 
close #6407
This was referenced Dec 8, 2022
Merged
Closed
This was referenced Dec 8, 2022
Closed
Closed
Closed
Closed
Closed
ti-chi-bot pushed a commit that referenced this issue Dec 9, 2022
@JaySon-Huang
PageStorage: background version compact for v2 (#6446) (#6447)
6076a21 
close #6407
ti-chi-bot added a commit that referenced this issue Jan 9, 2023
@ti-chi-bot
PageStorage: background version compact for v2 (#6446) (#6464)
6ca9e0d 
close #6407
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@JaySon-Huang JaySon-Huang

Labels
affects-5.0 affects-5.1 affects-5.2 affects-5.3 affects-5.4 This bug affects the 5.4.x(LTS) versions. affects-6.0 affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-6.5 This bug affects the 6.5.x(LTS) versions. component/storage severity/critical type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

PageStorage: background version compact for v2 JaySon-Huang/tiflash
5 participants
@flowbehappy @jebter @JaySon-Huang @VelocityLight @ti-chi-bot