Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

safekeeper: use jemalloc #9780

Merged
merged 4 commits into from
Nov 29, 2024
Merged

safekeeper: use jemalloc #9780

merged 4 commits into from
Nov 29, 2024

Conversation

erikgrinaker
Copy link
Contributor

@erikgrinaker erikgrinaker commented Nov 17, 2024

Problem

To add Safekeeper heap profiling in #9778, we need to switch to an allocator that supports it. Pageserver and proxy already use jemalloc.

Touches #9534.

Summary of changes

Use jemalloc in Safekeeper.

@erikgrinaker erikgrinaker requested a review from a team as a code owner November 17, 2024 10:44
@erikgrinaker erikgrinaker requested review from arpad-m and a team and removed request for a team November 17, 2024 10:44
Copy link

github-actions bot commented Nov 17, 2024

6952 tests run: 6644 passed, 0 failed, 308 skipped (full report)


Flaky tests (2)

Postgres 17

Postgres 15

Code coverage* (full report)

  • functions: 30.2% (8175 of 27038 functions)
  • lines: 47.6% (64710 of 135838 lines)

* collected from Rust tests only


The comment gets automatically updated with the latest test results
33b141d at 2024-11-29T13:27:39.853Z :recycle:

Base automatically changed from erik/jemalloc-0.6 to main November 17, 2024 18:52
@erikgrinaker erikgrinaker force-pushed the erik/safekeeper-jemalloc branch from 6e47b61 to 1646829 Compare November 17, 2024 19:10
@erikgrinaker
Copy link
Contributor Author

TFTR! I should run a few benchmarks before merging this, but don't expect any issues.

@erikgrinaker

This comment was marked as off-topic.

@erikgrinaker
Copy link
Contributor Author

erikgrinaker commented Nov 29, 2024

I added a bytes_reserve benchmark which emulates the allocations made when parsing inbound WAL messages. This shows a substantial improvement of up to 100% for larger allocations on Linux.

On macOS (not included here), jemalloc had roughly equivalent performance to the default allocator. This makes sense since they're both region-based allocators.

bytes_reserve/size=1    time:   [13.340 ns 13.402 ns 13.467 ns]
                        thrpt:  [70.817 MiB/s 71.160 MiB/s 71.491 MiB/s]
                 change:
                        time:   [-2.9146% -2.5058% -2.1043%] (p = 0.00 < 0.05)
                        thrpt:  [+2.1495% +2.5702% +3.0021%]

bytes_reserve/size=64   time:   [16.534 ns 16.572 ns 16.616 ns]
                        thrpt:  [3.5872 GiB/s 3.5967 GiB/s 3.6049 GiB/s]
                 change:
                        time:   [-8.4769% -8.3165% -8.1326%] (p = 0.00 < 0.05)
                        thrpt:  [+8.8525% +9.0708% +9.2620%]

bytes_reserve/size=1024 time:   [26.292 ns 26.673 ns 27.053 ns]
                        thrpt:  [35.252 GiB/s 35.755 GiB/s 36.273 GiB/s]
                 change:
                        time:   [-3.0939% -1.8861% -0.7759%] (p = 0.00 < 0.05)
                        thrpt:  [+0.7820% +1.9223% +3.1927%]

bytes_reserve/size=8192 time:   [69.892 ns 69.907 ns 69.923 ns]
                        thrpt:  [109.11 GiB/s 109.14 GiB/s 109.16 GiB/s]
                 change:
                        time:   [-49.603% -49.583% -49.560%] (p = 0.00 < 0.05)
                        thrpt:  [+98.256% +98.344% +98.426%]

bytes_reserve/size=131072
                        time:   [1.5824 µs 1.6067 µs 1.6337 µs]
                        thrpt:  [74.720 GiB/s 75.977 GiB/s 77.143 GiB/s]
                 change:
                        time:   [-18.309% -17.334% -16.544%] (p = 0.00 < 0.05)
                        thrpt:  [+19.823% +20.969% +22.412%]

@erikgrinaker erikgrinaker added this pull request to the merge queue Nov 29, 2024
Merged via the queue into main with commit a6073b5 Nov 29, 2024
80 checks passed
@erikgrinaker erikgrinaker deleted the erik/safekeeper-jemalloc branch November 29, 2024 13:38
github-merge-queue bot pushed a commit that referenced this pull request Dec 3, 2024
## Problem

We don't have good observability for memory usage. This would be useful
e.g. to debug OOM incidents or optimize performance or resource usage.

We would also like to use continuous profiling with e.g. [Grafana Cloud
Profiles](https://grafana.com/products/cloud/profiles-for-continuous-profiling/)
(see neondatabase/cloud#14888).

This PR is intended as a proof of concept, to try it out in staging and
drive further discussions about profiling more broadly.

Touches #9534.
Touches neondatabase/cloud#14888.
Depends on #9779.
Depends on #9780.

## Summary of changes

Adds a HTTP route `/profile/heap` that takes a heap profile and returns
it. Query parameters:

* `format`: output format (`jemalloc` or `pprof`; default `pprof`).

Unlike CPU profiles (see #9764), heap profiles are not symbolized and
require the original binary to translate addresses to function names. To
make this work with Grafana, we'll probably have to symbolize the
process server-side -- this is left as future work, as is other output
formats like SVG.

Heap profiles don't work on macOS due to limitations in jemalloc.
awarus pushed a commit that referenced this pull request Dec 5, 2024
## Problem

To add Safekeeper heap profiling in #9778, we need to switch to an
allocator that supports it. Pageserver and proxy already use jemalloc.

Touches #9534.

## Summary of changes

Use jemalloc in Safekeeper.
awarus pushed a commit that referenced this pull request Dec 5, 2024
## Problem

We don't have good observability for memory usage. This would be useful
e.g. to debug OOM incidents or optimize performance or resource usage.

We would also like to use continuous profiling with e.g. [Grafana Cloud
Profiles](https://grafana.com/products/cloud/profiles-for-continuous-profiling/)
(see neondatabase/cloud#14888).

This PR is intended as a proof of concept, to try it out in staging and
drive further discussions about profiling more broadly.

Touches #9534.
Touches neondatabase/cloud#14888.
Depends on #9779.
Depends on #9780.

## Summary of changes

Adds a HTTP route `/profile/heap` that takes a heap profile and returns
it. Query parameters:

* `format`: output format (`jemalloc` or `pprof`; default `pprof`).

Unlike CPU profiles (see #9764), heap profiles are not symbolized and
require the original binary to translate addresses to function names. To
make this work with Grafana, we'll probably have to symbolize the
process server-side -- this is left as future work, as is other output
formats like SVG.

Heap profiles don't work on macOS due to limitations in jemalloc.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants