CMake option to link jemalloc

[jiridanek]

http://jemalloc.net/

[jiridanek]

What to do with this? It might be best to go about it in three steps.

• First, add options to enable many possible alternative allocators in turn, easily, so that they can be tested with ease. This is important because allocators are tunable and tuning matters.
• Then, create abstraction over the common allocator extended API so that also the advanced features that go beyond malloc/free of each allocator can be tested (they include observability into what was allocated, and so on). This is necessary if the alloc_pool features are to be replaced with 3rd party allocator
• Finally, pick one allocator to build with, probably leave the abstraction in place, possibly even keep one or two alternative allocators for possibility of future retesting and reevaluation.

List of notable alternative allocators, from https://crates.io/crates/shadowsocks-rust build options

• jemalloc - Uses jemalloc as global memory allocator
• mimalloc - Uses mi-malloc as global memory allocator
• tcmalloc - Uses TCMalloc as global memory allocator. It tries to link system-wide tcmalloc by default, use vendored from source with tcmalloc-vendored.
• snmalloc - Uses snmalloc as global memory allocator
• rpmalloc - Uses rpmalloc as global memory allocator

tcmalloc is from google, jemalloc is facebook and openjdk, mimalloc is microsoft

• dlmalloc - honorable mention, debug focused (https://dmalloc.com)

[jiridanek]

Here's some negativity about jemalloc

• Rust used to use by default, but they backed away from it https://blog.rust-lang.org/2019/01/17/Rust-1.32.0.html#jemalloc-is-removed-by-default, here is a list of their issues caused by jemalloc Switch the default global allocator to System, remove alloc_jemalloc, use jemallocator in rustc rust-lang/rust#36963 (comment)
• One can do malloc_trim(0) which has benefits, https://www.joyfulbikeshedding.com/blog/2019-03-14-what-causes-ruby-memory-bloat.html#a-magic-trick-trimming, it returns pages back to the OS. They have a heap visualizer https://github.com/FooBarWidget/heap_dumper_visualizer. Discuss MALLOC_ARENA_MAX=2 https://devcenter.heroku.com/articles/tuning-glibc-memory-behavior, see also https://www.gnu.org/software/libc/manual/html_node/Malloc-Tunable-Parameters.html
• Fedora will stick to glibc, https://lwn.net/Articles/761502/