Alloc with MMAP

[accelerated]

Describe your changes

• Use MMAP instead of heap for coroutine allocations
• Moved header block at the end of the stack

The motivation behind using mmap() on Linux (and the equivalent feature on Windows - not yet supported) is to allow for immediate memory reclaiming of the coroutine stack (quite large). If heap allocations are used, although the allocated memory is freed, the dynamic memory management algorithm will not return immediately these blocks to the system as they may be reused at a later date (for obvious performance reasons). If the system is very busy, these blocks may become fragmented and new coroutine stack allocations will again require further expansion from system memory (via sbrk() or mmap()) as large contiguous areas won't be necessarily available. The result is that the application's RSS keeps increasing even through all coroutines may have finished processing.

mmap() is just as fast as a call to new() without the drawbacks of memory leakage. An alternative to using mmap() is to tinker with mallopt() M_MMAP_THRESHOLD and M_TRIM_THRESHOLD but it's not as reliable as mmap(). Furthermore, with mmap() we can now protect the lowest page of the coroutine stack (only for pre-allocated blocks) so that we may catch any stack overflows. These overflows are hard to debug and fail in very different (often confusing) ways.
One reason for not pre-allocating one, single large contiguous block for all pre-allocated coroutines (rather than piece-meal) is to prevent trampling from one stack to another. Even with page protection, a function may somehow write beyond the 4096 byte range of protection and hit the next stack. This is easily reproducible.

Testing performed
Ran entire test suite. Any coroutine allocation issues are sure to fail immediately.

Signed-off-by: Alexander Damian adamian@bloomberg.net

[arosenzweig3]

Looks good to me.