Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Document troubleshooting commands to check disk and memory health and performance #6859

Closed
shekhirin opened this issue Feb 28, 2024 · 3 comments · Fixed by #9364
Closed

Document troubleshooting commands to check disk and memory health and performance #6859

shekhirin opened this issue Feb 28, 2024 · 3 comments · Fixed by #9364
Assignees
@daobaniw
Labels
A-db Related to the database C-docs An addition or correction to our documentation D-good-first-issue Nice and easy! A great choice to get started

Comments

@shekhirin
Copy link
Collaborator

shekhirin commented Feb 28, 2024 •
edited
Loading

Describe the change

There is a bunch of tools users can run to help understand why the performance is degraded, if it's related to the hardware.

Reference numbers are from Latitude c3.large.x86.

  1. IOzone
    1. Test disk speed: iozone -e -t1 -i0 -i2 -r1k -s1g /tmp. Reference numbers: 
        Children see throughput for  1 initial writers  =  907733.81 kB/sec
  Parent sees throughput for  1 initial writers   =  907239.68 kB/sec
  Min throughput per process                      =  907733.81 kB/sec
  Max throughput per process                      =  907733.81 kB/sec
  Avg throughput per process                      =  907733.81 kB/sec
  Min xfer                                        = 1048576.00 kB

  Children see throughput for  1 rewriters        = 1765222.62 kB/sec
  Parent sees throughput for  1 rewriters         = 1763433.35 kB/sec
  Min throughput per process                      = 1765222.62 kB/sec
  Max throughput per process                      = 1765222.62 kB/sec
  Avg throughput per process                      = 1765222.62 kB/sec
  Min xfer                                        = 1048576.00 kB

  Children see throughput for 1 random readers    = 1557497.38 kB/sec
  Parent sees throughput for 1 random readers     = 1554846.58 kB/sec
  Min throughput per process                      = 1557497.38 kB/sec
  Max throughput per process                      = 1557497.38 kB/sec
  Avg throughput per process                      = 1557497.38 kB/sec
  Min xfer                                        = 1048576.00 kB

  Children see throughput for 1 random writers    =  984428.69 kB/sec
  Parent sees throughput for 1 random writers     =  983476.67 kB/sec
  Min throughput per process                      =  984428.69 kB/sec
  Max throughput per process                      =  984428.69 kB/sec
  Avg throughput per process                      =  984428.69 kB/sec
  Min xfer                                        = 1048576.00 kB
    2. Test disk speed with memory-mapped files: iozone -B -G -e -t1 -i0 -i2 -r1k -s1g /tmp. Reference numbers: 
        Children see throughput for  1 initial writers  =   56471.06 kB/sec
  Parent sees throughput for  1 initial writers   =   56365.14 kB/sec
  Min throughput per process                      =   56471.06 kB/sec
  Max throughput per process                      =   56471.06 kB/sec
  Avg throughput per process                      =   56471.06 kB/sec
  Min xfer                                        = 1048576.00 kB

  Children see throughput for  1 rewriters        =  241650.69 kB/sec
  Parent sees throughput for  1 rewriters         =  239067.96 kB/sec
  Min throughput per process                      =  241650.69 kB/sec
  Max throughput per process                      =  241650.69 kB/sec
  Avg throughput per process                      =  241650.69 kB/sec
  Min xfer                                        = 1048576.00 kB

  Children see throughput for 1 random readers    = 6833161.00 kB/sec
  Parent sees throughput for 1 random readers     = 5597659.65 kB/sec
  Min throughput per process                      = 6833161.00 kB/sec
  Max throughput per process                      = 6833161.00 kB/sec
  Avg throughput per process                      = 6833161.00 kB/sec
  Min xfer                                        = 1048576.00 kB

  Children see throughput for 1 random writers    =  220248.53 kB/sec
  Parent sees throughput for 1 random writers     =  219112.26 kB/sec
  Min throughput per process                      =  220248.53 kB/sec
  Max throughput per process                      =  220248.53 kB/sec
  Avg throughput per process                      =  220248.53 kB/sec
  Min xfer                                        = 1048576.00 kB
  2. Check RAM speed: sudo lshw -short -C memory. Reference numbers (see frequency): 
      H/W path              Device          Class          Description
  ================================================================
  /0/0                                  memory         64KiB BIOS
  /0/24                                 memory         512GiB System Memory
  /0/24/0                               memory         64GiB DIMM DDR4 Synchronous Registered (Buffered) 3200 MHz (0.3 ns)
  /0/24/1                               memory         64GiB DIMM DDR4 Synchronous Registered (Buffered) 3200 MHz (0.3 ns)
  /0/24/2                               memory         64GiB DIMM DDR4 Synchronous Registered (Buffered) 3200 MHz (0.3 ns)
  /0/24/3                               memory         64GiB DIMM DDR4 Synchronous Registered (Buffered) 3200 MHz (0.3 ns)
  /0/24/4                               memory         64GiB DIMM DDR4 Synchronous Registered (Buffered) 3200 MHz (0.3 ns)
  /0/24/5                               memory         64GiB DIMM DDR4 Synchronous Registered (Buffered) 3200 MHz (0.3 ns)
  /0/24/6                               memory         64GiB DIMM DDR4 Synchronous Registered (Buffered) 3200 MHz (0.3 ns)
  /0/24/7                               memory         64GiB DIMM DDR4 Synchronous Registered (Buffered) 3200 MHz (0.3 ns)
  /0/27                                 memory         1536KiB L1 cache
  /0/28                                 memory         12MiB L2 cache
  /0/29                                 memory         128MiB L3 cache
  3. Test RAM health: sudo memtester 10G (will take a while, test with sudo memtester 1G 1 first). All checks should report ok.

We need to document this in the Troubleshooting section of the book.

Additional context

No response
@shekhirin shekhirin added C-docs An addition or correction to our documentation D-good-first-issue Nice and easy! A great choice to get started A-db Related to the database labels Feb 28, 2024
@abhijeetbhagat
Copy link
Contributor

i can take this @shekhirin

@shekhirin
Copy link
Collaborator Author

@abhijeetbhagat amazing, thank you! Assigned you. Let me know if you have any questions.

@Rjected Rjected mentioned this issue Jun 24, 2024
Closed
1 task
@daobaniw
Copy link
Contributor

daobaniw commented Jul 8, 2024

Hi, I would like to take this.

@daobaniw daobaniw mentioned this issue Jul 8, 2024
Merged
@shekhirin shekhirin assigned daobaniw and unassigned abhijeetbhagat Jul 8, 2024
@github-project-automation github-project-automation bot moved this from Todo to Done in Reth Tracker Jul 8, 2024
@linear linear bot closed this as not planned Won't fix, can't repro, duplicate, stale Aug 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@daobaniw daobaniw

Labels
A-db Related to the database C-docs An addition or correction to our documentation D-good-first-issue Nice and easy! A great choice to get started
Projects
Reth Tracker
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

book: add troubleshooting commands to check disk and memory health and performance daobaniw/reth
3 participants
@shekhirin @abhijeetbhagat @daobaniw