Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

br node (8c27g) OOM when restore a big table of 150TB #51621

Closed
fubinzh opened this issue Mar 8, 2024 · 8 comments · Fixed by #51872
Closed

br node (8c27g) OOM when restore a big table of 150TB #51621

fubinzh opened this issue Mar 8, 2024 · 8 comments · Fixed by #51872
Labels
affects-8.0 component/br This issue is related to BR of TiDB. severity/major type/bug The issue is confirmed as a bug.

Comments

@fubinzh
Copy link

fubinzh commented Mar 8, 2024

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

  1. restore a 150TB big table on a 8c27g br node.

2. What did you expect to see? (Required)

br should not OOM.

3. What did you see instead (Required)

br OOM killed, stats is small for this backup.
image

image

4. What is your TiDB version? (Required)

master
@fubinzh fubinzh added the type/bug The issue is confirmed as a bug. label Mar 8, 2024
@fubinzh
Copy link
Author

fubinzh commented Mar 8, 2024

/component br

@ti-chi-bot ti-chi-bot bot added the component/br This issue is related to BR of TiDB. label Mar 8, 2024
@fubinzh
Copy link
Author

fubinzh commented Mar 8, 2024

/severity major

@fubinzh
Copy link
Author

fubinzh commented Mar 11, 2024

Simliar OOM seen when restoring a backup of 100TB (~3k+ tables) using 16c64g br node with enviroment variable GOMEMLIMIT=32GiB.

@3pointer
Copy link
Contributor

#51777 fixed it

@3pointer 3pointer reopened this Mar 18, 2024
@fubinzh
Copy link
Author

fubinzh commented Mar 25, 2024

This issue is still seen when restoring a backup of 30TB(~500k tables) using 32c110g br node with enviroment variable GOMEMLIMIT=16GiB, br parameters: "--granularity=coarse-grained","--tikv-max-restore-concurrency=128", "--checksum=false", "--load-stats=true", "--ratelimit=600"
image
image
image

@3pointer 3pointer mentioned this issue Mar 26, 2024
Merged
13 tasks
@fubinzh
Copy link
Author

fubinzh commented Mar 26, 2024

/remove-label may-affects-6.1

@fubinzh
Copy link
Author

fubinzh commented Mar 26, 2024

/remove-label may-affects-6.5
/remove-label may-affects-7.1
/remove-label may-affects-7.5

@BornChanger
Copy link
Contributor

/remove-label may-affects-5.4

@ti-chi-bot ti-chi-bot bot removed the may-affects-5.4 This bug maybe affects 5.4.x versions. label Mar 26, 2024
ti-chi-bot bot pushed a commit that referenced this issue Mar 26, 2024
@3pointer
restore: use a limited workpool (#51872)
b6dd179 
ref #50701, close #51621
@ti-chi-bot ti-chi-bot mentioned this issue Mar 26, 2024
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Mar 26, 2024
@ti-chi-bot
restore: use a limited workpool (#51872) (#52110)
d4ef3b2 
ref #50701, close #51621
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
affects-8.0 component/br This issue is related to BR of TiDB. severity/major type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

restore: use a limited workpool 3pointer/tidb
4 participants
@jebter @3pointer @fubinzh @BornChanger