Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hello WebAssembly! (MVP implementation) #10870

Merged
merged 48 commits into from
Mar 21, 2022
Merged
Changes from 26 commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
cabdb42
Add Wasm32 LibC (based on musl)
lbguilherme Jul 1, 2021
76d5e22
Add Basic Wasm32 support
lbguilherme Jul 1, 2021
98c5447
Add LibC as a symbolic link
lbguilherme Jul 1, 2021
94109db
Merge branch 'master' into feat/webassembly
lbguilherme Jul 14, 2021
fa5077a
Move wasm to a Crystal::System implementation
lbguilherme Jul 14, 2021
fe511de
Enable smoke tests for wasm32-wasi
lbguilherme Jul 14, 2021
165b974
Revert change on unix Thread
lbguilherme Jul 14, 2021
be594ed
Smoke test: build compiler for the wasm test
lbguilherme Jul 14, 2021
ffe4b5b
Merge remote-tracking branch 'upstream/master' into feat/webassembly
lbguilherme Jul 20, 2021
598c99f
Mark stub methods with NotImplementedError or TODO
lbguilherme Jul 21, 2021
2d78c9b
Update src/crystal/system/wasm/process.cr
lbguilherme Jul 21, 2021
ffd9c12
Update wasi libc: it's now limited to what wasi actually provides. Mo…
lbguilherme Jul 22, 2021
9faf715
Use arc4random instead of getentropy
lbguilherme Jul 22, 2021
b0797c1
Remove Signal, Process and Sockets unimplemented method from Wasm32
lbguilherme Aug 2, 2021
aec74ad
Work around compiler bug in previous commit: BUG: `typeof(_io)` at ex…
lbguilherme Aug 2, 2021
463d4cd
fix: cleanup from previous commit
lbguilherme Aug 2, 2021
5faf601
Merge remote-tracking branch 'upstream/master' into feat/webassembly
lbguilherme Aug 2, 2021
3d91387
Merge remote-tracking branch 'upstream/master' into feat/webassembly
lbguilherme Aug 3, 2021
b1dd5f7
crystal tool format
lbguilherme Aug 3, 2021
00b10d6
fix: use correct .wasm extension
lbguilherme Dec 18, 2021
eab4739
Merge remote-tracking branch 'upstream/master' into feat/webassembly
lbguilherme Dec 18, 2021
4e33d92
Merge remote-tracking branch 'upstream/master' into feat/webassembly
lbguilherme Dec 23, 2021
e86b675
feat: implement Crystal::System::Fiber with malloc/free
lbguilherme Dec 23, 2021
ba5a531
fix: mark system_linger as not implemented
lbguilherme Dec 24, 2021
2582817
feat: add direct wasi integration for Dir.entries
lbguilherme Dec 24, 2021
4d5b061
feat: enable libpcre
lbguilherme Dec 24, 2021
5b4627f
chore: refactor Crystal::Program.object_extension
lbguilherme Dec 28, 2021
2519a6a
fix: mock Thread.current
lbguilherme Dec 28, 2021
fdef3ac
chore: change PREOPENS to @@preopens
lbguilherme Dec 28, 2021
10c6eb7
fix: print fatal errors
lbguilherme Dec 28, 2021
a0407ca
fix: use Sync logger until channels work
lbguilherme Dec 28, 2021
c3f035f
chore: rename wasierror.cr -> wasi_error.cr
lbguilherme Dec 28, 2021
3ba1b96
chore: preopen cleanup
lbguilherme Dec 28, 2021
e58cf26
fix: there is no need to initialize PREOPENS as early as possible if …
lbguilherme Dec 29, 2021
7bcd75a
fix: missing wasi require
lbguilherme Dec 29, 2021
cb2ea23
feat: generate better linker command
lbguilherme Dec 29, 2021
ad2e402
chore: stub Exception::CallStack#decode_backtrace until callback support
lbguilherme Dec 29, 2021
2e4b517
chore: remove some unused pieces from wasm libc
lbguilherme Dec 31, 2021
5faa13a
fix: use "out fd"
lbguilherme Jan 27, 2022
9de1150
feat: add wasi random_get to avoid relying on arc4random polyfill
lbguilherme Feb 8, 2022
2d3158e
Mark WASI as a UNIX target and use it as the system environment, inst…
lbguilherme Feb 8, 2022
ac74bad
Update spec/compiler/loader/unix_spec.cr
lbguilherme Feb 8, 2022
9f72ef9
Update spec/compiler/ffi/ffi_spec.cr
lbguilherme Feb 8, 2022
fea289a
Revert "Update spec/compiler/loader/unix_spec.cr"
lbguilherme Feb 8, 2022
53f6c25
Revert "Update spec/compiler/ffi/ffi_spec.cr"
lbguilherme Feb 8, 2022
5995375
fix: correct usage of some wasm32<->wasi flags
lbguilherme Feb 8, 2022
570eb32
Merge remote-tracking branch 'upstream/master' into feat/webassembly
lbguilherme Feb 11, 2022
b011761
fix: home_path for wasi
lbguilherme Feb 11, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .github/workflows/smoke.yml
Original file line number Diff line number Diff line change
@@ -46,6 +46,7 @@ jobs:
- aarch64-darwin
- arm-linux-gnueabihf
- i386-linux-musl
- wasm32-unknown-wasi
- i386-linux-gnu
- x86_64-dragonfly
- x86_64-freebsd
@@ -56,5 +57,9 @@ jobs:
- name: Download Crystal source
uses: actions/checkout@v2

- name: Make compiler (special case only for wasm32, until next release).
if: ${{ matrix.target == 'wasm32-unknown-wasi' }}
run: bin/ci with_build_env make

- name: Run smoke test
run: bin/ci with_build_env make smoke_test target=${{ matrix.target }}
2 changes: 2 additions & 0 deletions spec/std/process_spec.cr
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
{% skip_file if flag?(:wasm32) %}

require "spec"
require "process"
require "./spec_helper"
2 changes: 2 additions & 0 deletions spec/std/signal_spec.cr
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
{% skip_file if flag?(:wasm32) %}

require "spec"
require "signal"

4 changes: 2 additions & 2 deletions spec/std/socket/addrinfo_spec.cr
Original file line number Diff line number Diff line change
@@ -74,8 +74,8 @@ describe Socket::Addrinfo, tags: "network" do
end

describe "Error" do
{% unless flag?(:win32) %}
# This method is not available on windows because windows support was introduced after deprecation.
{% unless flag?(:win32) || flag?(:wasm32) %}
# This method is not available on windows/wasm because windows/wasm support was introduced after deprecation.
it ".new (deprecated)" do
error = Socket::Addrinfo::Error.new(LibC::EAI_NONAME, "No address found", "foobar.com")
error.os_error.should eq Errno.new(LibC::EAI_NONAME)
2 changes: 2 additions & 0 deletions spec/std/socket/tcp_server_spec.cr
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
{% skip_file if flag?(:wasm32) %}

require "./spec_helper"
require "../../support/win32"

2 changes: 2 additions & 0 deletions spec/std/socket/tcp_socket_spec.cr
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
{% skip_file if flag?(:wasm32) %}

require "./spec_helper"
require "../../support/win32"

12 changes: 7 additions & 5 deletions spec/std/string_spec.cr
Original file line number Diff line number Diff line change
@@ -2715,12 +2715,14 @@ describe "String" do
string.should be(clone)
end

it "allocates buffer of correct size when UInt8 is given to new (#3332)" do
String.new(255_u8) do |buffer|
LibGC.size(buffer).should be >= 255
{255, 0}
{% unless flag?(:wasm32) %}
it "allocates buffer of correct size when UInt8 is given to new (#3332)" do
String.new(255_u8) do |buffer|
LibGC.size(buffer).should be >= 255
{255, 0}
end
end
end
{% end %}

it "raises on String.new if returned bytesize is greater than capacity" do
expect_raises ArgumentError, "Bytesize out of capacity bounds" do
2 changes: 1 addition & 1 deletion src/compiler/crystal/codegen/link.cr
Original file line number Diff line number Diff line change
@@ -100,7 +100,7 @@ module Crystal

class Program
def object_extension
has_flag?("windows") ? ".obj" : ".o"
has_flag?("windows") ? ".obj" : has_flag?("wasm32") ? ".wasm" : ".o"
lbguilherme marked this conversation as resolved.
Show resolved Hide resolved
end

def lib_flags
2 changes: 2 additions & 0 deletions src/compiler/crystal/codegen/target.cr
Original file line number Diff line number Diff line change
@@ -144,6 +144,8 @@ class Crystal::Codegen::Target
if cpu.empty? && !features.includes?("fp") && armhf?
features += "+vfp2"
end
when "wasm32"
LLVM.init_webassembly
else
raise Target::Error.new("Unsupported architecture for target triple: #{self}")
end
2 changes: 1 addition & 1 deletion src/compiler/crystal/command.cr
Original file line number Diff line number Diff line change
@@ -247,7 +247,7 @@ class Crystal::Command
begin
elapsed = Time.measure do
Process.run(output_filename, args: run_args, input: Process::Redirect::Inherit, output: Process::Redirect::Inherit, error: Process::Redirect::Inherit) do |process|
{% unless flag?(:win32) %}
{% unless flag?(:win32) || flag?(:wasm32) %}
# Ignore the signal so we don't exit the running process
# (the running process can still handle this signal)
::Signal::INT.ignore # do
3 changes: 3 additions & 0 deletions src/crystal/system.cr
Original file line number Diff line number Diff line change
@@ -19,6 +19,9 @@ end
{% elsif flag?(:win32) %}
require "./system/win32/hostname"
require "./system/win32/cpucount"
{% elsif flag?(:wasm32) %}
require "./system/wasm/hostname"
require "./system/wasm/cpucount"
{% else %}
{% raise "No Crystal::System implementation available" %}
{% end %}
2 changes: 2 additions & 0 deletions src/crystal/system/dir.cr
Original file line number Diff line number Diff line change
@@ -53,6 +53,8 @@ end

{% if flag?(:unix) %}
require "./unix/dir"
{% elsif flag?(:wasm32) %}
require "./wasm/dir"
lbguilherme marked this conversation as resolved.
Show resolved Hide resolved
{% elsif flag?(:win32) %}
require "./win32/dir"
{% else %}
6 changes: 4 additions & 2 deletions src/crystal/system/env.cr
Original file line number Diff line number Diff line change
@@ -12,8 +12,10 @@ module Crystal::System::Env
# def self.each(&block : String, String ->)
end

{% if flag?(:win32) %}
{% if flag?(:unix) || flag?(:wasm32) %}
require "./unix/env"
{% elsif flag?(:win32) %}
require "./win32/env"
{% else %}
require "./unix/env"
{% raise "No implementation of Crystal::System::Env available" %}
{% end %}
4 changes: 3 additions & 1 deletion src/crystal/system/event_loop.cr
Original file line number Diff line number Diff line change
@@ -28,7 +28,9 @@ struct Crystal::Event
# def add(time_span : Time::Span?) : Nil
end

{% if flag?(:unix) %}
{% if flag?(:wasm32) %}
require "./wasm/event_loop"
{% elsif flag?(:unix) %}
require "./unix/event_loop_libevent"
{% elsif flag?(:win32) %}
require "./win32/event_loop_iocp"
2 changes: 2 additions & 0 deletions src/crystal/system/fiber.cr
Original file line number Diff line number Diff line change
@@ -13,6 +13,8 @@ end
require "./unix/fiber"
{% elsif flag?(:win32) %}
require "./win32/fiber"
{% elsif flag?(:wasm32) %}
require "./wasm/fiber"
{% else %}
{% raise "fiber not supported" %}
{% end %}
2 changes: 2 additions & 0 deletions src/crystal/system/file.cr
Original file line number Diff line number Diff line change
@@ -53,6 +53,8 @@ end
require "./unix/file"
{% elsif flag?(:win32) %}
require "./win32/file"
{% elsif flag?(:wasm32) %}
require "./wasm/file"
{% else %}
{% raise "No Crystal::System::File implementation available" %}
{% end %}
8 changes: 6 additions & 2 deletions src/crystal/system/file_descriptor.cr
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
{% if flag?(:win32) %}
{% if flag?(:unix) %}
require "./unix/file_descriptor"
{% elsif flag?(:win32) %}
require "./win32/file_descriptor"
{% elsif flag?(:wasm32) %}
require "./wasm/file_descriptor"
{% else %}
require "./unix/file_descriptor"
{% raise "No Crystal::System::FileDescriptor implementation available" %}
{% end %}
2 changes: 1 addition & 1 deletion src/crystal/system/file_info.cr
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{% if flag?(:unix) %}
{% if flag?(:unix) || flag?(:wasm32) %}
require "./unix/file_info"
{% elsif flag?(:win32) %}
require "./win32/file_info"
2 changes: 2 additions & 0 deletions src/crystal/system/group.cr
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
{% if flag?(:unix) %}
require "./unix/group"
{% elsif flag?(:wasm32) %}
require "./wasm/group"
{% else %}
{% raise "No Crystal::System::Group implementation available" %}
{% end %}
2 changes: 1 addition & 1 deletion src/crystal/system/mime.cr
Original file line number Diff line number Diff line change
@@ -3,7 +3,7 @@ module Crystal::System::MIME
# def self.load
end

{% if flag?(:unix) %}
{% if flag?(:unix) || flag?(:wasm32) %}
require "./unix/mime"
{% elsif flag?(:win32) %}
require "./win32/mime"
2 changes: 1 addition & 1 deletion src/crystal/system/path.cr
Original file line number Diff line number Diff line change
@@ -3,7 +3,7 @@ module Crystal::System::Path
# def self.home : String
end

{% if flag?(:unix) %}
{% if flag?(:unix) || flag?(:wasm32) %}
require "./unix/path"
{% elsif flag?(:win32) %}
require "./win32/path"
2 changes: 2 additions & 0 deletions src/crystal/system/process.cr
Original file line number Diff line number Diff line change
@@ -74,6 +74,8 @@ end
require "./unix/process"
{% elsif flag?(:win32) %}
require "./win32/process"
{% elsif flag?(:wasm32) %}
require "./wasm/process"
{% else %}
{% raise "No Crystal::System::Process implementation available" %}
{% end %}
2 changes: 1 addition & 1 deletion src/crystal/system/random.cr
Original file line number Diff line number Diff line change
@@ -12,7 +12,7 @@ end

{% if flag?(:linux) %}
require "./unix/getrandom"
{% elsif flag?(:openbsd) || flag?(:netbsd) %}
{% elsif flag?(:openbsd) || flag?(:netbsd) || flag?(:wasm32) %}
require "./unix/arc4random"
{% elsif flag?(:unix) %}
require "./unix/urandom"
2 changes: 2 additions & 0 deletions src/crystal/system/socket.cr
Original file line number Diff line number Diff line change
@@ -78,6 +78,8 @@ end

{% if flag?(:unix) %}
require "./unix/socket"
{% elsif flag?(:wasm32) %}
require "./wasm/socket"
{% elsif flag?(:win32) %}
require "./win32/socket"
{% else %}
2 changes: 2 additions & 0 deletions src/crystal/system/thread.cr
Original file line number Diff line number Diff line change
@@ -33,6 +33,8 @@ require "./thread_linked_list"
require "./unix/pthread_condition_variable"
{% elsif flag?(:win32) %}
require "./win32/thread"
{% elsif flag?(:wasm32) %}
require "./wasm/thread"
{% else %}
{% raise "thread not supported" %}
{% end %}
2 changes: 2 additions & 0 deletions src/crystal/system/thread_mutex.cr
Original file line number Diff line number Diff line change
@@ -21,6 +21,8 @@ end
require "./unix/pthread_mutex"
{% elsif flag?(:win32) %}
require "./win32/thread_mutex"
{% elsif flag?(:wasm32) %}
require "./wasm/thread_mutex"
{% else %}
{% raise "thread not supported" %}
{% end %}
6 changes: 4 additions & 2 deletions src/crystal/system/time.cr
Original file line number Diff line number Diff line change
@@ -12,8 +12,10 @@ module Crystal::System::Time
# def self.load_localtime : ::Time::Location?
end

{% if flag?(:win32) %}
{% if flag?(:unix) || flag?(:wasm32) %}
require "./unix/time"
{% elsif flag?(:win32) %}
require "./win32/time"
{% else %}
require "./unix/time"
{% raise "No implementation of Crystal::System::Time available" %}
{% end %}
2 changes: 1 addition & 1 deletion src/crystal/system/unix/arc4random.cr
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{% skip_file unless flag?(:openbsd) || flag?(:netbsd) %}
{% skip_file unless flag?(:openbsd) || flag?(:netbsd) || flag?(:wasm32) %}

require "c/stdlib"

2 changes: 2 additions & 0 deletions src/crystal/system/user.cr
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
{% if flag?(:unix) %}
require "./unix/user"
{% elsif flag?(:wasm32) %}
require "./wasm/user"
{% else %}
{% raise "No Crystal::System::User implementation available" %}
{% end %}
6 changes: 6 additions & 0 deletions src/crystal/system/wasm/cpucount.cr
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
module Crystal::System
def self.cpu_count
# TODO: There isn't a good way to get the number of CPUs on WebAssembly
1
end
end
105 changes: 105 additions & 0 deletions src/crystal/system/wasm/dir.cr
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
module Crystal::System::Dir
private class DirHandle
property fd : LibWasi::Fd
property buf = Bytes.new(4096)
property pos = 4096u32
property end_pos = 4096u32
property cookie = 0u64

def initialize(@fd)
end

def fill_buffer(path)
err = LibWasi.fd_readdir(@fd, @buf, @buf.size, @cookie, pointerof(@end_pos))
raise ::File::Error.from_os_error("Error reading directory entries", err, file: path) unless err.success?
@pos = 0
end
end

def self.open(path : String) : DirHandle
parent_fd, relative_path = Wasi.find_path_preopen(path)

fd = uninitialized LibWasi::Fd
err = LibWasi.path_open(parent_fd, LibWasi::LookupFlags::SymlinkFollow, relative_path, LibWasi::OpenFlags::Directory, LibWasi::Rights::FdReaddir, LibWasi::Rights::None, LibWasi::FdFlags::None, pointerof(fd))
lbguilherme marked this conversation as resolved.
Show resolved Hide resolved
raise ::File::Error.from_os_error("Error opening directory", err, file: path) unless err.success?

DirHandle.new(fd)
end

def self.next_entry(dir, path) : Entry?
if dir.end_pos < dir.buf.size && dir.pos >= dir.end_pos
return nil
end

if dir.pos + sizeof(LibWasi::DirEnt) > dir.buf.size
dir.fill_buffer(path)
end

dirent = Pointer(LibWasi::DirEnt).new(dir.buf.to_unsafe.address + dir.pos).value

if dir.pos + sizeof(LibWasi::DirEnt) + dirent.d_namlen > dir.buf.size
if dir.pos == 0
dir.buf = Bytes.new(dir.buf.size * 2)
end
dir.fill_buffer(path)
return next_entry(dir, path)
end

name = String.new(dir.buf[dir.pos + sizeof(LibWasi::DirEnt), dirent.d_namlen])
dir.pos += sizeof(LibWasi::DirEnt) + dirent.d_namlen
dir.cookie = dirent.d_next

is_dir = case dirent.d_type
when .directory? then true
when .unknown?, .symbolic_link? then nil
else false
end

Entry.new(name, is_dir)
end

def self.rewind(dir) : Nil
dir.cookie = 0
dir.end_pos = dir.pos = dir.buf.size.to_u32
end

def self.close(dir, path) : Nil
err = LibWasi.fd_close(dir.fd)
raise ::File::Error.from_os_error("Error opening directory", err, file: path) unless err.success?
end

def self.current : String
unless dir = LibC.getcwd(nil, 0)
raise ::File::Error.from_errno("Error getting current directory", file: "./")
end

dir_str = String.new(dir)
LibC.free(dir.as(Void*))
dir_str
end

def self.current=(path : String)
if LibC.chdir(path.check_no_null_byte) != 0
raise ::File::Error.from_errno("Error while changing directory", file: path)
end

path
end

def self.tempdir
tmpdir = ENV["TMPDIR"]? || "/tmp"
tmpdir.rchop(::File::SEPARATOR)
end

def self.create(path : String, mode : Int32) : Nil
if LibC.mkdir(path.check_no_null_byte, mode) == -1
raise ::File::Error.from_errno("Unable to create directory", file: path)
end
end

def self.delete(path : String) : Nil
if LibC.rmdir(path.check_no_null_byte) == -1
raise ::File::Error.from_errno("Unable to remove directory", file: path)
end
end
end
Loading