Auto merge of #45905 - alexcrichton:add-wasm-target, r=aturon

std: Add a new wasm32-unknown-unknown target

This commit adds a new target to the compiler: wasm32-unknown-unknown. This target is a reimagining of what it looks like to generate WebAssembly code from Rust. Instead of using Emscripten which can bring with it a weighty runtime this instead is a target which uses only the LLVM backend for WebAssembly and a "custom linker" for now which will hopefully one day be direct calls to lld.

Notable features of this target include:

* There is zero runtime footprint. The target assumes nothing exists other than the wasm32 instruction set.
* There is zero toolchain footprint beyond adding the target. No custom linker is needed, rustc contains everything.
* Very small wasm modules can be generated directly from Rust code using this target.
* Most of the standard library is stubbed out to return an error, but anything related to allocation works (aka `HashMap`, `Vec`, etc).
* Naturally, any `#[no_std]` crate should be 100% compatible with this new target.

This target is currently somewhat janky due to how linking works. The "linking" is currently unconditional whole program LTO (aka LLVM is being used as a linker). Naturally that means compiling programs is pretty slow! Eventually though this target should have a linker.

This target is also intended to be quite experimental. I'm hoping that this can act as a catalyst for further experimentation in Rust with WebAssembly. Breaking changes are very likely to land to this target, so it's not recommended to rely on it in any critical capacity yet. We'll let you know when it's "production ready".

### Building yourself

First you'll need to configure the build of LLVM and enable this target

```
$ ./configure --target=wasm32-unknown-unknown --set llvm.experimental-targets=WebAssembly
```

Next you'll want to remove any previously compiled LLVM as it needs to be rebuilt with WebAssembly support. You can do that with:

```
$ rm -rf build
```

And then you're good to go! A `./x.py build` should give you a rustc with the appropriate libstd target.

### Test support

Currently testing-wise this target is looking pretty good but isn't complete. I've got almost the entire `run-pass` test suite working with this target (lots of tests ignored, but many passing as well). The `core` test suite is [still getting LLVM bugs fixed](https://reviews.llvm.org/D39866) to get that working and will take some time. Relatively simple programs all seem to work though!

In general I've only tested this with a local fork that makes use of LLVM 5 rather than our current LLVM 4 on master. The LLVM 4 WebAssembly backend AFAIK isn't broken per se but is likely missing bug fixes available on LLVM 5. I'm hoping though that we can decouple the LLVM 5 upgrade and adding this wasm target!

### But the modules generated are huge!

It's worth nothing that you may not immediately see the "smallest possible wasm module" for the input you feed to rustc. For various reasons it's very difficult to get rid of the final "bloat" in vanilla rustc (again, a real linker should fix all this). For now what you'll have to do is:

    cargo install --git https://github.com/alexcrichton/wasm-gc
    wasm-gc foo.wasm bar.wasm

And then `bar.wasm` should be the smallest we can get it!

---

In any case for now I'd love feedback on this, particularly on the various integration points if you've got better ideas of how to approach them!

.gitmodules

@@ -42,3 +42,9 @@
 [submodule "src/tools/miri"]
 	path = src/tools/miri
 	url = https://github.com/solson/miri.git
+[submodule "src/dlmalloc"]
+	path = src/dlmalloc
+	url = https://github.com/alexcrichton/dlmalloc-rs.git
+[submodule "src/binaryen"]
+	path = src/binaryen
+	url = https://github.com/alexcrichton/binaryen.git

src/bootstrap/check.rs

@@ -1211,7 +1211,8 @@ impl Step for Crate {
                     // ends up messing with various mtime calculations and such.
                     if !name.contains("jemalloc") &&
                        *name != *"build_helper" &&
-                       !(name.starts_with("rustc_") && name.ends_with("san")) {
+                       !(name.starts_with("rustc_") && name.ends_with("san")) &&
+                       name != "dlmalloc" {
                         cargo.arg("-p").arg(&format!("{}:0.0.0", name));
                     }
                     for dep in build.crates[&name].deps.iter() {

src/bootstrap/dist.rs

@@ -672,6 +672,9 @@ fn copy_src_dirs(build: &Build, src_dirs: &[&str], exclude_dirs: &[&str], dst_di
              spath.ends_with(".s")) {
             return false
         }
+        if spath.contains("test/emscripten") || spath.contains("test\\emscripten") {
+            return false
+        }
 
         let full_path = Path::new(dir).join(path);
         if exclude_dirs.iter().any(|excl| full_path == Path::new(excl)) {
@@ -736,6 +739,7 @@ impl Step for Src {
         // (essentially libstd and all of its path dependencies)
         let std_src_dirs = [
             "src/build_helper",
+            "src/dlmalloc",
             "src/liballoc",
             "src/liballoc_jemalloc",
             "src/liballoc_system",
@@ -754,6 +758,7 @@ impl Step for Src {
             "src/libunwind",
             "src/rustc/compiler_builtins_shim",
             "src/rustc/libc_shim",
+            "src/rustc/dlmalloc_shim",
             "src/libtest",
             "src/libterm",
             "src/jemalloc",

src/etc/wasm32-shim.js

@@ -0,0 +1,119 @@
+// Copyright 2017 The Rust Project Developers. See the COPYRIGHT
+// file at the top-level directory of this distribution and at
+// http://rust-lang.org/COPYRIGHT.
+//
+// Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
+// http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your
+// option. This file may not be copied, modified, or distributed
+// except according to those terms.
+
+// This is a small "shim" program which is used when wasm32 unit tests are run
+// in this repository. This program is intended to be run in node.js and will
+// load a wasm module into memory, instantiate it with a set of imports, and
+// then run it.
+//
+// There's a bunch of helper functions defined here in `imports.env`, but note
+// that most of them aren't actually needed to execute most programs. Many of
+// these are just intended for completeness or debugging. Hopefully over time
+// nothing here is needed for completeness.
+
+const fs = require('fs');
+const process = require('process');
+const buffer = fs.readFileSync(process.argv[2]);
+
+Error.stackTraceLimit = 20;
+
+let m = new WebAssembly.Module(buffer);
+
+let memory = null;
+
+function copystr(a, b) {
+  if (memory === null) {
+    return null
+  }
+  let view = new Uint8Array(memory.buffer).slice(a, a + b);
+  return String.fromCharCode.apply(null, view);
+}
+
+let imports = {};
+imports.env = {
+  // These are generated by LLVM itself for various intrinsic calls. Hopefully
+  // one day this is not necessary and something will automatically do this.
+  fmod: function(x, y) { return x % y; },
+  exp2: function(x) { return Math.pow(2, x); },
+  exp2f: function(x) { return Math.pow(2, x); },
+  ldexp: function(x, y) { return x * Math.pow(2, y); },
+  ldexpf: function(x, y) { return x * Math.pow(2, y); },
+  log10: function(x) { return Math.log10(x); },
+  log10f: function(x) { return Math.log10(x); },
+
+  // These are called in src/libstd/sys/wasm/stdio.rs and are used when
+  // debugging is enabled.
+  rust_wasm_write_stdout: function(a, b) {
+    let s = copystr(a, b);
+    if (s !== null) {
+      process.stdout.write(s);
+    }
+  },
+  rust_wasm_write_stderr: function(a, b) {
+    let s = copystr(a, b);
+    if (s !== null) {
+      process.stderr.write(s);
+    }
+  },
+
+  // These are called in src/libstd/sys/wasm/args.rs and are used when
+  // debugging is enabled.
+  rust_wasm_args_count: function() {
+    if (memory === null)
+      return 0;
+    return process.argv.length - 2;
+  },
+  rust_wasm_args_arg_size: function(i) {
+    return process.argv[i + 2].length;
+  },
+  rust_wasm_args_arg_fill: function(idx, ptr) {
+    let arg = process.argv[idx + 2];
+    let view = new Uint8Array(memory.buffer);
+    for (var i = 0; i < arg.length; i++) {
+      view[ptr + i] = arg.charCodeAt(i);
+    }
+  },
+
+  // These are called in src/libstd/sys/wasm/os.rs and are used when
+  // debugging is enabled.
+  rust_wasm_getenv_len: function(a, b) {
+    let key = copystr(a, b);
+    if (key === null) {
+      return -1;
+    }
+    if (!(key in process.env)) {
+      return -1;
+    }
+    return process.env[key].length;
+  },
+  rust_wasm_getenv_data: function(a, b, ptr) {
+    let key = copystr(a, b);
+    let value = process.env[key];
+    let view = new Uint8Array(memory.buffer);
+    for (var i = 0; i < value.length; i++) {
+      view[ptr + i] = value.charCodeAt(i);
+    }
+  },
+};
+
+let module_imports = WebAssembly.Module.imports(m);
+
+for (var i = 0; i < module_imports.length; i++) {
+  let imp = module_imports[i];
+  if (imp.module != 'env') {
+    continue
+  }
+  if (imp.name == 'memory' && imp.kind == 'memory') {
+    memory = new WebAssembly.Memory({initial: 20});
+    imports.env.memory = memory;
+  }
+}
+
+let instance = new WebAssembly.Instance(m, imports);

src/liballoc_jemalloc/build.rs

@@ -31,7 +31,7 @@ fn main() {
     let host = env::var("HOST").expect("HOST was not set");
     if target.contains("rumprun") || target.contains("bitrig") || target.contains("openbsd") ||
        target.contains("msvc") || target.contains("emscripten") || target.contains("fuchsia") ||
-       target.contains("redox") {
+       target.contains("redox") || target.contains("wasm32") {
         println!("cargo:rustc-cfg=dummy_jemalloc");
         return;
     }

src/liballoc_system/Cargo.toml

@@ -13,3 +13,7 @@ doc = false
 alloc = { path = "../liballoc" }
 core = { path = "../libcore" }
 libc = { path = "../rustc/libc_shim" }
+
+# See comments in the source for what this dependency is
+[target.'cfg(all(target_arch = "wasm32", not(target_os = "emscripten")))'.dependencies]
+dlmalloc = { path = "../rustc/dlmalloc_shim" }

src/liballoc_system/lib.rs

@@ -34,12 +34,14 @@
               target_arch = "powerpc64",
               target_arch = "asmjs",
               target_arch = "wasm32")))]
+#[allow(dead_code)]
 const MIN_ALIGN: usize = 8;
 #[cfg(all(any(target_arch = "x86_64",
               target_arch = "aarch64",
               target_arch = "mips64",
               target_arch = "s390x",
               target_arch = "sparc64")))]
+#[allow(dead_code)]
 const MIN_ALIGN: usize = 16;
 
 extern crate alloc;
@@ -458,3 +460,91 @@ mod platform {
         }
     }
 }
+
+// This is an implementation of a global allocator on the wasm32 platform when
+// emscripten is not in use. In that situation there's no actual runtime for us
+// to lean on for allocation, so instead we provide our own!
+//
+// The wasm32 instruction set has two instructions for getting the current
+// amount of memory and growing the amount of memory. These instructions are the
+// foundation on which we're able to build an allocator, so we do so! Note that
+// the instructions are also pretty "global" and this is the "global" allocator
+// after all!
+//
+// The current allocator here is the `dlmalloc` crate which we've got included
+// in the rust-lang/rust repository as a submodule. The crate is a port of
+// dlmalloc.c from C to Rust and is basically just so we can have "pure Rust"
+// for now which is currently technically required (can't link with C yet).
+//
+// The crate itself provides a global allocator which on wasm has no
+// synchronization as there are no threads!
+#[cfg(all(target_arch = "wasm32", not(target_os = "emscripten")))]
+mod platform {
+    extern crate dlmalloc;
+
+    use alloc::heap::{Alloc, AllocErr, Layout, Excess, CannotReallocInPlace};
+    use System;
+    use self::dlmalloc::GlobalDlmalloc;
+
+    #[unstable(feature = "allocator_api", issue = "32838")]
+    unsafe impl<'a> Alloc for &'a System {
+        #[inline]
+        unsafe fn alloc(&mut self, layout: Layout) -> Result<*mut u8, AllocErr> {
+            GlobalDlmalloc.alloc(layout)
+        }
+
+        #[inline]
+        unsafe fn alloc_zeroed(&mut self, layout: Layout)
+            -> Result<*mut u8, AllocErr>
+        {
+            GlobalDlmalloc.alloc_zeroed(layout)
+        }
+
+        #[inline]
+        unsafe fn dealloc(&mut self, ptr: *mut u8, layout: Layout) {
+            GlobalDlmalloc.dealloc(ptr, layout)
+        }
+
+        #[inline]
+        unsafe fn realloc(&mut self,
+                          ptr: *mut u8,
+                          old_layout: Layout,
+                          new_layout: Layout) -> Result<*mut u8, AllocErr> {
+            GlobalDlmalloc.realloc(ptr, old_layout, new_layout)
+        }
+
+        #[inline]
+        fn usable_size(&self, layout: &Layout) -> (usize, usize) {
+            GlobalDlmalloc.usable_size(layout)
+        }
+
+        #[inline]
+        unsafe fn alloc_excess(&mut self, layout: Layout) -> Result<Excess, AllocErr> {
+            GlobalDlmalloc.alloc_excess(layout)
+        }
+
+        #[inline]
+        unsafe fn realloc_excess(&mut self,
+                                 ptr: *mut u8,
+                                 layout: Layout,
+                                 new_layout: Layout) -> Result<Excess, AllocErr> {
+            GlobalDlmalloc.realloc_excess(ptr, layout, new_layout)
+        }
+
+        #[inline]
+        unsafe fn grow_in_place(&mut self,
+                                ptr: *mut u8,
+                                layout: Layout,
+                                new_layout: Layout) -> Result<(), CannotReallocInPlace> {
+            GlobalDlmalloc.grow_in_place(ptr, layout, new_layout)
+        }
+
+        #[inline]
+        unsafe fn shrink_in_place(&mut self,
+                                  ptr: *mut u8,
+                                  layout: Layout,
+                                  new_layout: Layout) -> Result<(), CannotReallocInPlace> {
+            GlobalDlmalloc.shrink_in_place(ptr, layout, new_layout)
+        }
+    }
+}

src/libcore/Cargo.toml

@@ -9,9 +9,6 @@ path = "lib.rs"
 test = false
 bench = false
 
-[dev-dependencies]
-rand = "0.3"
-
 [[test]]
 name = "coretests"
 path = "../libcore/tests/lib.rs"

src/libcore/tests/lib.rs

@@ -28,7 +28,6 @@
 #![feature(iter_rfind)]
 #![feature(iter_rfold)]
 #![feature(nonzero)]
-#![feature(rand)]
 #![feature(raw)]
 #![feature(refcell_replace_swap)]
 #![feature(sip_hash_13)]
@@ -49,7 +48,6 @@
 
 extern crate core;
 extern crate test;
-extern crate rand;
 
 mod any;
 mod array;