RLBox v2 RFC

[deian]

RLBox v2 RFC

RLBox is a C++ framework for sandboxing third party C libraries within the same process. We designed the framework to assist engineers and partially automate the hard parts of in-process sandboxing: retrofitting the application code to securely interface with the now-untrusted library. RLBox is used in Firefox to isolate the Graphite, Hunspell, Ogg, Expat, and Woof2 libraries on all supported Firefox platforms (desktop and mobile). (See the Mozilla hacks post if you haven't already!)

To make this happen we had to evolve both RLBox and the underlying sandboxing toolchain---our fork of wasm2c---to improve both usability and performance. But there is still lots of room for improvement and we're now working on RLBox version 2. The goal of this document is to describe our plan for v2 (and some of the design decisions) and iterate on a framework that is useful to the broader community beyond Firefox. At a high level, the plan is to:

1. Add support for high-level memory management across the sandbox boundary.
2. Add support for sandboxing C++ libraries.
3. Make sandboxes relocatable (and reduce memory footprint in turn).
4. Upstream RLBox changes to wasm2c.
5. Add SIMD support to wasm2c.

We're very open to feedback and want to support more use cases. So, if you're using RLBox or are starting to look at library sandboxing, we'd love to collaborate---drop us a note!

RLBox-v2 design

Eliminate (manual memory management) boilerplate

We plan to reduce the amount of boilerplate code developers need to write in order to sandbox libraries. This boilerplate code is required today because RLBox exposes the sandboxed library as a foreign function interface (FFI), and like all FFIs, this is exposed as a C interface. This, in turn, means:

• The data structures that are shared between Firefox and the sandboxed library are managed manually, using low-level interfaces (in the style of C's malloc, memcpy, and free), and

• RLBox requires sandboxed libraries to expose a C-style interface for all public functions and data structures in the sandboxed library. For a C++ application like Firefox, this means that developers need to add boilerplate code to Firefox (to ensure shared data structures are managed using C-style primitives), as well as the library (to ensure all functions and data structures are exposed using a C-style interface).

Consider, for example, a snippet from Firefox where we invoke the sandboxed Expat XML parsing library:

// namespace separator
const char16_t kSep[] = {0xFFFF, '\0'};
const char16_t kUTF16[] = ... ;

nsresult nsExpatDriver::Initialize(...) {
  // Create sandbox with pre-declared size.
  // Sandbox size is set using domain specific info:
  // => XML parsing occurs in 2MB chunks
  const uint64_t sandboxSize = 2 * 1024 * 1024;
  mSbx = rlbox::create_sandbox(sandboxSize);
  // Create expat parser, copy the encoding and
  // namespace separator into the sandbox.
  auto sep = mSbx->malloc(/* len */);
  rlbox::memcpy(mSbx, sep, kSep, /* len */);
  auto utf16 = mSbx->malloc(/* len */);
  rlbox::memcpy(mSbx, utf16, kUTF16, /* len */);
  ...
  mExpatParser = mSbx->invoke(
    MOZ_XML_ParserCreate_MM, *utf16, NULL, *sep);
  ...
  mSbx->free(sep);
  mSbx->free(utf16);
}

Here, Firefox calls the Expat MOZ_XML_ParserCreate_MM that accepts two strings and returns an XML parser object. (You can ignore the MOZ_ prefix.) These strings have to be transferred into the sandbox memory for the sandboxed Expat to do its job. Today, we do this use malloc to allocate the strings, memcpy to copy them into the sandbox and, later, free to deallocate these strings.

NOTE: These aren't exactly libc's malloc, memcpy, and free. The RLBox versions actually do a few things under the hood, including automatically inserting sandboxing security checks and handling ABI differences between Firefox and the sandboxed library. As user-facing APIs though, they're (deliberately) like the C standard library APIs.

The problem with such code is that it's error-prone (manual memory management always is) and potentially expensive (e.g., we sometimes copy buffer across the sandbox boundary when we could have moved them).

Proposal

To reduce the amount of boilerplate code, we propose to support a new high-level C++ interface for sandboxed libraries.

High-level memory management

RLBox-v2 will provide higher-level APIs such as TransferBuffer that both provide a vocabulary for common operations and use C++'s smart pointers such as unique_ptr to simplify memory management. This will allow us to rewrite the above code as:

// namespace separator
const char16_t kSep[] = {0xFFFF, '\0'};
const char16_t kUTF16[] = ... ;

nsresult nsExpatDriver::Initialize(...) {
  // Create sandbox
  mSbx = rlbox::create_sandbox();
  // Create expat parser, copy the encoding and
  // namespace separator into the sandbox.
  auto sep = TransferBuffer(mSbx, kSep, /* len */);
  auto utf16 = TransferBuffer(mSbx, kUTF16, /* len */);
  ...
  mExpatParser = mSbx->invoke(
    MOZ_XML_ParserCreate_MM, *utf16, NULL, *sep);
  ...
}

Internally, TransferBuffer allocates, copies, and frees the (application-side) buffer automatically. We experimented with such a TransferBuffer interface in Firefox when sandboxing Woff2 (see e.g., this code); our plan for v2 is to integrate these into RLBox itself and use standard C++ unique pointers.

We will also add APIs for other common operations such as sharing of stack variables and safe iteration over buffers in sandbox memory.

Support for C++ libraries

We will add support for C++ libraries. In particular, we will also allow RLBox-v2 to directly call C++ functions in, and share C++ data structures with the sandbox. We plant do this by modifying RLBox to auto-generate C wrappers for libraries---tedious boilerplate code that developers have to write by hand today. For instance, when sandboxing hunspell, a spellchecking library used by Firefox, we could not directly invoke hunspell's APIs as they are written in C++ and use the C++ string type std::string; rather, we had to write C wrappers for these functions that we could invoke instead. RLBox-v2's approach of generating C wrappers gives us a simple, portable way to support C++, a task that is generally challenging for FFI utilities as C++ does not have a standard ABI that can be used across all platforms and compilers.

Support small, relocatable sandboxes

We plan to reduce the memory footprint of Wasm-based sandboxing so that developers don't have to manually account for memory pressure due to fragmentation. This is particularly important on machines using 32-bit architectures---they have a limited virtual memory and thus more sensitive to fragmentation. In the wild, we've seen multiple instances of Firefox's sandboxed libraries running out of memory due to fragmentation, and crashing the browser.

Fundamentally, the problem is that RLBox currently requires the Wasm sandbox's memory---a contiguous region of memory that is used as the heap for sandboxed code---to remain in a single location through the lifetime of the sandbox. This restriction, imposes constraints on the growth of the sandbox's memory; specifically, all growth of the sandbox memory must be done in-place without migrating the memory to a new location.

In practice, sandboxed libraries in Firefox deal with this by computing the maximum expected memory size and pre-emptively reserving this memory to ensure memory fragmentation doesn't restrict growth of the sandbox. While this approach works, it significantly increases the memory footprint of sandboxed code---and thus results in instances where Firefox OOMs.

Proposal

To address the above challenges, we plan to add support for relocatable sandboxes in RLBox-v2. This would allow sandboxes to start with a small memory size that is expandable on demand, and relocatable if fragmentation prevents growth of memory in the current location.

Relocatable sandboxes, however, require a change in the implementation of one of RLBox's core features: the automatic handling of ABI differences. Wasm sandboxed code has a number of ABI differences from the host application including differences in the pointer representation. Specifically, Wasm sandboxes use pointers that are represented as relative offsets from the start of the sandbox memory, while applications uses the standard absolute pointers. This difference in ABI is currently handled by RLBox's typesystem automatically by marking all relative pointers returned from the sandbox with a unique type and converting them to absolute pointers when they are exposed to application code. However, such a conversion may fail for relocatable sandboxes when the sandbox memory is relocated after a pointer has been converted.

In RLBox-v2, we plan to support relocatable sandboxes by changing RLBox to always store pointers as relative offsets on 32-bit systems, and convert them each time they are used. Specifically, pointers would be represented in the structure shown below rather than the simple global pointers:

struct rlboxv2-pointer{
  char** sandbox_memory_location_ref;
  uint32_t ptr_as_relative_offset;
};

There are two main design constrains we want to uphold as we move to this approach:

1. Our implementations for memory constrained devices should now slow down the general implementation that is optimized for the majority of machines. We propose to address this by allowing RLBox to easily switch between pointer representations based on the resources of the device target.

2. Our implementation must handle the scenario where the sandbox memory is being relocated concurrently with a different application thread accessing the sandbox memory. Rather than solve this with expensive locks, we propose to address this using reference counting to delay any deallocations of sandbox memory (that occur as part of relocation) until all references to this memory have been discarded.

Extend and optimize (upstream) wasm2c for library sandboxing

RLBox can be used with different sandboxing toolchains. What we ultimately settled on in Firefox is wasm2c, largely because of it portability across different OS and CPU architectures.¹

To use wasm2c in Firefox, we had to add several features to make it suitable for library sandboxing. We did this in our fork of wasm2c, which extends wasm2c with (1) features specific to library sandboxing (e.g., support for callback functions) and (2) several performance and debugging features, which are useful beyond the library sandboxing use case. Our features have been tested and deployed in production Firefox for 9+ months without incident. Our plan is to upstream these changes to wasm2c (and kudos to Sam Clegg, Alon Zakai and Keith Winstein for their help). This ill allow us to standardize on a well-supported, maintained, and standard toolchain for sandboxing.

We're also working on adding SIMD support to wasm2c; without SIMD we can't sandbox certain libraries (e.g., libvps and libSoundTouch) without giving up on performance. This, too, is in the works using SIMD-everywhere, a library that provides conversions from Wasm SIMD intrinsics to the native platform intrinsics.

Footnotes

1. wasm2c compiles Wasm to ANSI C which, for all its faults, can be easily compiled to a binary targeting any platform or architecture. ↩

[shravanrn]

Following some discussions, we will also likely implement the segue optimization in wasm2c. Specifically we will be using x64's (limited support) for segmentation as an optimization for performance
See writeup and benchmarks here https://plas2022.github.io/files/pdf/SegueColorGuard.pdf

Per our internal performance benchmarks wasm2c is tied as the fastest Wasm AOT compiler with WAMR, WAVM etc. (the one possibly faster Wasm engine is awsm, but this needs further investigation to confirm). A few external benchmarks also report essentially the same results. The latest one I've seen is https://00f.net/2023/01/04/webassembly-benchmark-2023/

With this segue optimization, wasm2c would likely be one of the fastest wasm implementation available.