Performance pass for ralloc: global-local allocator model, TLS alloca…

…tor, and more

- Ralloc now has a global-local allocator model where the local allocator requests memory from the global allocator. This allows for...

- ...thread-local storage allocators. The local allocator stores the allocator in a TLS static, bypassing the mutex.

- Changes to OOM handling: the OOM handleris no longer allocator-specific, but instead global

- Thread-local OOMs

- Tweak canonicalization.

- Add UniCell primive, which allow for single-threaded mutexes

- Tests and stuff.

README.md

@@ -57,6 +57,23 @@ fn main() {
 }
 ```
 
+### Thread-specific OOM handlers.
+
+You can override the global OOM handler for your current thread. Enable the `thread_oom` feature, and then do:
+
+```rust
+extern crate ralloc;
+
+fn my_handler() -> ! {
+    println!("Oh no. Blame the Mexicans.");
+}
+
+fn main() {
+    ralloc::set_thread_oom_handler(my_handler);
+    // Do some stuff...
+}
+```
+
 ### Debug check: double free
 
 Ooh, this one is a cool one. `ralloc` detects various memory bugs when compiled
@@ -87,21 +104,20 @@ fn main() {
 ```rust
 extern crate ralloc;
 
-use std::mem;
+use std::{mem, spawn};
 
 fn main() {
-    {
-        // We start by allocating some stuff.
-        let a = Box::new(500u32);
-        // We then leak `a`.
-        let b = mem::forget(a);
-    }
-    // The box is now leaked, and the destructor won't be called.
-
-    // To debug this we insert a memory leak check in the end of our programs.
-    // This will panic if a memory leak is found (and will be a NOOP without
-    // `debug_tools`).
-    ralloc::lock().debug_assert_no_leak();
+    thread::spawn(|| {
+        {
+            // We start by allocating some stuff.
+            let a = Box::new(500u32);
+            // We then leak `a`.
+            let b = mem::forget(a);
+        }
+        // The box is now leaked, and the destructor won't be called.
+
+        // When this thread exits, the program will panic.
+    });
 }
 ```
 
@@ -227,13 +243,8 @@ This is just one of many examples.
 ### Platform agnostic
 
 `ralloc` is platform independent. It depends on `ralloc_shim`, a minimal
-interface for platform dependent functions. The default implementation of
-`ralloc_shim` requires the following symbols:
-
-1. `sbrk`: For extending the data segment size.
-2. `sched_yield`: For the spinlock.
-3. `memcpy`, `memcmp`, `memset`: Core memory routines.
-4. `rust_begin_unwind`: For panicking.
+interface for platform dependent functions. An default implementation of
+`ralloc_shim` is provided (supporting Mac OS, Linux, and BSD).
 
 ### Local allocators
 
@@ -268,7 +279,7 @@ fn main() {
 
 ### Logging
 
-If you enable the `log` feature, you get detailed locking of the allocator, e.g.
+If you enable the `log` feature, you get detailed logging of the allocator, e.g.
 
 ```
 |   : BRK'ing a block of size, 80, and alignment 8.            (at bookkeeper.rs:458)

TODO.md

@@ -1,5 +1,7 @@
 - [x] Thread local allocator.
 - [x] Lock reuse
+- [ ] Freeze/unfreeze -- memcompression
+- [ ] Proper error messages in unwraps.
 - [ ] Checkpoints
 - [ ] Fast `calloc`
 - [ ] Microcaches.

shim/src/lib.rs

@@ -1,21 +1,83 @@
-//! Symbols and externs that ralloc depends on.
+//! Symbols and externs that `ralloc` depends on.
+//!
+//! This crate provides implementation/import of these in Linux, BSD, and Mac OS.
 
-#![crate_name="ralloc_shim"]
-#![crate_type="lib"]
-#![feature(lang_items)]
-#![warn(missing_docs)]
+#![feature(lang_items, linkage)]
 #![no_std]
+#![warn(missing_docs)]
+
+extern crate libc;
+
+pub use libc::sched_yield;
+
+extern {
+    /// Change the data segment. See `man sbrk`.
+    pub fn sbrk(libc::intptr_t) -> *const libc::c_void;
+}
+
+/// Thread destructors for Linux.
+#[cfg(target_os = "linux")]
+pub mod thread_destructor {
+    use libc;
+
+    extern {
+        #[linkage = "extern_weak"]
+        static __dso_handle: *mut u8;
+        #[linkage = "extern_weak"]
+        static __cxa_thread_atexit_impl: *const libc::c_void;
+    }
+
+    /// Does this platform support thread destructors?
+    ///
+    /// This will return true, if and only if `__cxa_thread_atexit_impl` is non-null.
+    #[inline]
+    pub fn is_supported() -> bool {
+        !__cxa_thread_atexit_impl.is_null()
+    }
 
-extern "C" {
-    /// Cooperatively gives up a timeslice to the OS scheduler.
-    pub fn sched_yield() -> isize;
+    /// Register a thread destructor.
+    ///
+    /// # Safety
+    ///
+    /// This is unsafe due to accepting (and dereferencing) raw pointers, as well as running an
+    /// arbitrary unsafe function.
+    ///
+    /// On older system without the `__cxa_thread_atexit_impl` symbol, this is unsafe to call, and will
+    /// likely segfault.
+    // TODO: Due to rust-lang/rust#18804, make sure this is not generic!
+    pub unsafe fn register(t: *mut u8, dtor: unsafe extern fn(*mut u8)) {
+        use core::mem;
+
+        /// A thread destructor.
+        type Dtor = unsafe extern fn(dtor: unsafe extern fn(*mut u8), arg: *mut u8, dso_handle: *mut u8) -> libc::c_int;
+
+        mem::transmute::<*const libc::c_void, Dtor>(__cxa_thread_atexit_impl)(dtor, t, &__dso_handle as *const _ as *mut _);
+    }
+}
+
+/// Thread destructors for Mac OS.
+#[cfg(target_os = "macos")]
+pub mod thread_destructor {
+    use libc;
 
-    /// Increment data segment of this process by some, _n_, return a pointer to the new data segment
-    /// start.
+    /// Does this platform support thread destructors?
     ///
-    /// This uses the system call BRK as backend.
+    /// This will always return true.
+    #[inline]
+    pub fn is_supported() -> bool { true }
+
+    /// Register a thread destructor.
     ///
-    /// This is unsafe for multiple reasons. Most importantly, it can create an inconsistent state,
-    /// because it is not atomic. Thus, it can be used to create Undefined Behavior.
-    pub fn sbrk(n: isize) -> *mut u8;
+    /// # Safety
+    ///
+    /// This is unsafe due to accepting (and dereferencing) raw pointers, as well as running an
+    /// arbitrary unsafe function.
+    #[cfg(target_os = "macos")]
+    pub unsafe fn register(t: *mut u8, dtor: unsafe extern fn(*mut u8)) {
+        extern {
+            fn _tlv_atexit(dtor: unsafe extern fn(*mut u8), arg: *mut u8);
+        }
+
+        _tlv_atexit(dtor, t);
+    }
 }

src/allocator.rs

@@ -4,16 +4,30 @@
 
 use prelude::*;
 
-use sync;
+use {sync, breaker};
 use bookkeeper::Bookkeeper;
 
 /// The global default allocator.
-static ALLOCATOR: sync::Mutex<Allocator> = sync::Mutex::new(Allocator::new());
+static GLOBAL_ALLOCATOR: sync::Mutex<Allocator<breaker::Sbrk>> = sync::Mutex::new(Allocator::new());
+tls! {
+    /// The thread-local allocator.
+    static ALLOCATOR: Option<UniCell<Allocator<breaker::Global>>> = None;
+}
 
-/// Lock the allocator.
+/// Get the allocator.
 #[inline]
-pub fn lock<'a>() -> sync::MutexGuard<'a, Allocator> {
-    ALLOCATOR.lock()
+pub fn get() -> Result<Allocator<breaker::Global>, ()> {
+    if ALLOCATOR.is_none() {
+        // Create the new allocator.
+        let mut alloc = Allocator::new();
+        // Attach the allocator to the current thread.
+        alloc.attach();
+
+        // To get mutable access, we wrap it in an `UniCell`.
+        ALLOCATOR = Some(UniCell::new(alloc));
+
+        &ALLOCATOR
+    }
 }
 
 /// An allocator.
@@ -90,15 +104,4 @@ impl Allocator {
             Err(())
         }
     }
-
-    /// Assert that no leaks are done.
-    ///
-    /// This should be run in the end of your program, after destructors have been run. It will then
-    /// panic if some item is not freed.
-    ///
-    /// In release mode, this is a NOOP.
-    pub fn debug_assert_no_leak(&self) {
-        #[cfg(feature = "debug_tools")]
-        self.inner.assert_no_leak();
-    }
 }

src/block.rs

@@ -267,6 +267,13 @@ impl fmt::Debug for Block {
     }
 }
 
+/// Make sure dropped blocks are empty.
+impl Drop for Block {
+    fn drop(&mut self) {
+        debug_assert!(self.is_empty(), "Dropping a non-empty block.");
+    }
+}
+
 #[cfg(test)]
 mod test {
     use prelude::*;