-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement the module linking proposal in Wasmtime #2094
Labels
wasmtime
Issues about wasmtime that don't fall into another label
Comments
alexcrichton
added
the
wasmtime
Issues about wasmtime that don't fall into another label
label
Aug 4, 2020
It seems like this is done? |
Indeed! |
alexcrichton
added a commit
to alexcrichton/wasmtime
that referenced
this issue
Nov 23, 2020
This commit implements the interpretation necessary of the instance section of the module linking proposal. Instantiating a module which itself has nested instantiated instances will now instantiate the nested instances properly. This isn't all that useful without the ability to alias exports off the result, but we can at least observe the side effects of instantiation through the `start` function. cc bytecodealliance#2094
alexcrichton
added a commit
to alexcrichton/wasmtime
that referenced
this issue
Nov 23, 2020
This commit implements the interpretation necessary of the instance section of the module linking proposal. Instantiating a module which itself has nested instantiated instances will now instantiate the nested instances properly. This isn't all that useful without the ability to alias exports off the result, but we can at least observe the side effects of instantiation through the `start` function. cc bytecodealliance#2094
This was referenced Nov 23, 2020
alexcrichton
added a commit
to alexcrichton/wasmtime
that referenced
this issue
Nov 24, 2020
This commit implements the interpretation necessary of the instance section of the module linking proposal. Instantiating a module which itself has nested instantiated instances will now instantiate the nested instances properly. This isn't all that useful without the ability to alias exports off the result, but we can at least observe the side effects of instantiation through the `start` function. cc bytecodealliance#2094
alexcrichton
added a commit
to alexcrichton/wasmtime
that referenced
this issue
Nov 25, 2020
This commit is intended to do almost everything necessary for processing the alias section of module linking. Most of this is internal refactoring, the highlights being: * Type contents are now stored separately from a `wasmtime_env::Module`. Given that modules can freely alias types and have them used all over the place, it seemed best to have one canonical location to type storage which everywhere else points to (with indices). A new `TypeTables` structure is produced during compilation which is shared amongst all member modules in a wasm blob. * Instantiation is heavily refactored to account for module linking. The main gotcha here is that imports are now listed as "initializers". We have a sort of pseudo-bytecode-interpreter which interprets the initialization of a module. This is more complicated than just matching imports at this point because in the module linking proposal the module, alias, import, and instance sections may all be interleaved. This means that imports aren't guaranteed to show up at the beginning of the address space for modules/instances. Otherwise most of the changes here largely fell out from these two design points. Aliases are recorded as initializers in this scheme. Copying around type information and/or just knowing type information during compilation is also pretty easy since everything is just a pointer into a `TypeTables` and we don't have to actually copy any types themselves. Lots of various refactorings were necessary to accomodate these changes. Tests are hoped to cover a breadth of functionality here, but not necessarily a depth. There's still one more piece of the module linking proposal missing which is exporting instances/modules, which will come in a future PR. It's also worth nothing that there's one large TODO which isn't implemented in this change that I plan on opening an issue for. With module linking when a set of modules comes back from compilation each modules has all the trampolines for the entire set of modules. This is quite a lot of duplicate trampolines across module-linking modules. We'll want to refactor this at some point to instead have only one set of trampolines per set of module linking modules and have them shared from there. I figured it was best to separate out this change, however, since it's purely related to resource usage, and doesn't impact non-module-linking modules at all. cc bytecodealliance#2094
alexcrichton
added a commit
to alexcrichton/wasmtime
that referenced
this issue
Nov 25, 2020
This commit is intended to do almost everything necessary for processing the alias section of module linking. Most of this is internal refactoring, the highlights being: * Type contents are now stored separately from a `wasmtime_env::Module`. Given that modules can freely alias types and have them used all over the place, it seemed best to have one canonical location to type storage which everywhere else points to (with indices). A new `TypeTables` structure is produced during compilation which is shared amongst all member modules in a wasm blob. * Instantiation is heavily refactored to account for module linking. The main gotcha here is that imports are now listed as "initializers". We have a sort of pseudo-bytecode-interpreter which interprets the initialization of a module. This is more complicated than just matching imports at this point because in the module linking proposal the module, alias, import, and instance sections may all be interleaved. This means that imports aren't guaranteed to show up at the beginning of the address space for modules/instances. Otherwise most of the changes here largely fell out from these two design points. Aliases are recorded as initializers in this scheme. Copying around type information and/or just knowing type information during compilation is also pretty easy since everything is just a pointer into a `TypeTables` and we don't have to actually copy any types themselves. Lots of various refactorings were necessary to accomodate these changes. Tests are hoped to cover a breadth of functionality here, but not necessarily a depth. There's still one more piece of the module linking proposal missing which is exporting instances/modules, which will come in a future PR. It's also worth nothing that there's one large TODO which isn't implemented in this change that I plan on opening an issue for. With module linking when a set of modules comes back from compilation each modules has all the trampolines for the entire set of modules. This is quite a lot of duplicate trampolines across module-linking modules. We'll want to refactor this at some point to instead have only one set of trampolines per set of module linking modules and have them shared from there. I figured it was best to separate out this change, however, since it's purely related to resource usage, and doesn't impact non-module-linking modules at all. cc bytecodealliance#2094
alexcrichton
added a commit
that referenced
this issue
Dec 1, 2020
This commit implements the interpretation necessary of the instance section of the module linking proposal. Instantiating a module which itself has nested instantiated instances will now instantiate the nested instances properly. This isn't all that useful without the ability to alias exports off the result, but we can at least observe the side effects of instantiation through the `start` function. cc #2094
alexcrichton
added a commit
to alexcrichton/wasmtime
that referenced
this issue
Dec 1, 2020
This commit is intended to do almost everything necessary for processing the alias section of module linking. Most of this is internal refactoring, the highlights being: * Type contents are now stored separately from a `wasmtime_env::Module`. Given that modules can freely alias types and have them used all over the place, it seemed best to have one canonical location to type storage which everywhere else points to (with indices). A new `TypeTables` structure is produced during compilation which is shared amongst all member modules in a wasm blob. * Instantiation is heavily refactored to account for module linking. The main gotcha here is that imports are now listed as "initializers". We have a sort of pseudo-bytecode-interpreter which interprets the initialization of a module. This is more complicated than just matching imports at this point because in the module linking proposal the module, alias, import, and instance sections may all be interleaved. This means that imports aren't guaranteed to show up at the beginning of the address space for modules/instances. Otherwise most of the changes here largely fell out from these two design points. Aliases are recorded as initializers in this scheme. Copying around type information and/or just knowing type information during compilation is also pretty easy since everything is just a pointer into a `TypeTables` and we don't have to actually copy any types themselves. Lots of various refactorings were necessary to accomodate these changes. Tests are hoped to cover a breadth of functionality here, but not necessarily a depth. There's still one more piece of the module linking proposal missing which is exporting instances/modules, which will come in a future PR. It's also worth nothing that there's one large TODO which isn't implemented in this change that I plan on opening an issue for. With module linking when a set of modules comes back from compilation each modules has all the trampolines for the entire set of modules. This is quite a lot of duplicate trampolines across module-linking modules. We'll want to refactor this at some point to instead have only one set of trampolines per set of module linking modules and have them shared from there. I figured it was best to separate out this change, however, since it's purely related to resource usage, and doesn't impact non-module-linking modules at all. cc bytecodealliance#2094
alexcrichton
added a commit
to alexcrichton/wasmtime
that referenced
this issue
Dec 1, 2020
This commit implements the final piece of the module linking proposal which is to flesh out the support for importing/exporting instances and modules. This ended up having a few changes: * Two more `PrimaryMap` instances are now stored in an `Instance`. The value for instances is `InstanceHandle` (pretty easy) and for modules it's `Box<dyn Any>` (less easy). * The custom host state for `InstanceHandle` for `wasmtime` is now `Arc<TypeTables` to be able to fully reconstruct an instance's types just from its instance. * Type matching for imports now has been updated to take instances/modules into account. One of the main downsides of this implementation is that type matching of imports is duplicated between wasmparser and wasmtime, leading to posssible bugs especially in the subtelties of module linking. I'm not sure how best to unify these two pieces of validation, however, and it may be more trouble than it's worth. cc bytecodealliance#2094
alexcrichton
added a commit
to alexcrichton/wasmtime
that referenced
this issue
Dec 2, 2020
This commit is intended to do almost everything necessary for processing the alias section of module linking. Most of this is internal refactoring, the highlights being: * Type contents are now stored separately from a `wasmtime_env::Module`. Given that modules can freely alias types and have them used all over the place, it seemed best to have one canonical location to type storage which everywhere else points to (with indices). A new `TypeTables` structure is produced during compilation which is shared amongst all member modules in a wasm blob. * Instantiation is heavily refactored to account for module linking. The main gotcha here is that imports are now listed as "initializers". We have a sort of pseudo-bytecode-interpreter which interprets the initialization of a module. This is more complicated than just matching imports at this point because in the module linking proposal the module, alias, import, and instance sections may all be interleaved. This means that imports aren't guaranteed to show up at the beginning of the address space for modules/instances. Otherwise most of the changes here largely fell out from these two design points. Aliases are recorded as initializers in this scheme. Copying around type information and/or just knowing type information during compilation is also pretty easy since everything is just a pointer into a `TypeTables` and we don't have to actually copy any types themselves. Lots of various refactorings were necessary to accomodate these changes. Tests are hoped to cover a breadth of functionality here, but not necessarily a depth. There's still one more piece of the module linking proposal missing which is exporting instances/modules, which will come in a future PR. It's also worth nothing that there's one large TODO which isn't implemented in this change that I plan on opening an issue for. With module linking when a set of modules comes back from compilation each modules has all the trampolines for the entire set of modules. This is quite a lot of duplicate trampolines across module-linking modules. We'll want to refactor this at some point to instead have only one set of trampolines per set of module linking modules and have them shared from there. I figured it was best to separate out this change, however, since it's purely related to resource usage, and doesn't impact non-module-linking modules at all. cc bytecodealliance#2094
alexcrichton
added a commit
to alexcrichton/wasmtime
that referenced
this issue
Dec 2, 2020
This commit is intended to do almost everything necessary for processing the alias section of module linking. Most of this is internal refactoring, the highlights being: * Type contents are now stored separately from a `wasmtime_env::Module`. Given that modules can freely alias types and have them used all over the place, it seemed best to have one canonical location to type storage which everywhere else points to (with indices). A new `TypeTables` structure is produced during compilation which is shared amongst all member modules in a wasm blob. * Instantiation is heavily refactored to account for module linking. The main gotcha here is that imports are now listed as "initializers". We have a sort of pseudo-bytecode-interpreter which interprets the initialization of a module. This is more complicated than just matching imports at this point because in the module linking proposal the module, alias, import, and instance sections may all be interleaved. This means that imports aren't guaranteed to show up at the beginning of the address space for modules/instances. Otherwise most of the changes here largely fell out from these two design points. Aliases are recorded as initializers in this scheme. Copying around type information and/or just knowing type information during compilation is also pretty easy since everything is just a pointer into a `TypeTables` and we don't have to actually copy any types themselves. Lots of various refactorings were necessary to accomodate these changes. Tests are hoped to cover a breadth of functionality here, but not necessarily a depth. There's still one more piece of the module linking proposal missing which is exporting instances/modules, which will come in a future PR. It's also worth nothing that there's one large TODO which isn't implemented in this change that I plan on opening an issue for. With module linking when a set of modules comes back from compilation each modules has all the trampolines for the entire set of modules. This is quite a lot of duplicate trampolines across module-linking modules. We'll want to refactor this at some point to instead have only one set of trampolines per set of module linking modules and have them shared from there. I figured it was best to separate out this change, however, since it's purely related to resource usage, and doesn't impact non-module-linking modules at all. cc bytecodealliance#2094
alexcrichton
added a commit
to alexcrichton/wasmtime
that referenced
this issue
Dec 2, 2020
This commit is intended to do almost everything necessary for processing the alias section of module linking. Most of this is internal refactoring, the highlights being: * Type contents are now stored separately from a `wasmtime_env::Module`. Given that modules can freely alias types and have them used all over the place, it seemed best to have one canonical location to type storage which everywhere else points to (with indices). A new `TypeTables` structure is produced during compilation which is shared amongst all member modules in a wasm blob. * Instantiation is heavily refactored to account for module linking. The main gotcha here is that imports are now listed as "initializers". We have a sort of pseudo-bytecode-interpreter which interprets the initialization of a module. This is more complicated than just matching imports at this point because in the module linking proposal the module, alias, import, and instance sections may all be interleaved. This means that imports aren't guaranteed to show up at the beginning of the address space for modules/instances. Otherwise most of the changes here largely fell out from these two design points. Aliases are recorded as initializers in this scheme. Copying around type information and/or just knowing type information during compilation is also pretty easy since everything is just a pointer into a `TypeTables` and we don't have to actually copy any types themselves. Lots of various refactorings were necessary to accomodate these changes. Tests are hoped to cover a breadth of functionality here, but not necessarily a depth. There's still one more piece of the module linking proposal missing which is exporting instances/modules, which will come in a future PR. It's also worth nothing that there's one large TODO which isn't implemented in this change that I plan on opening an issue for. With module linking when a set of modules comes back from compilation each modules has all the trampolines for the entire set of modules. This is quite a lot of duplicate trampolines across module-linking modules. We'll want to refactor this at some point to instead have only one set of trampolines per set of module linking modules and have them shared from there. I figured it was best to separate out this change, however, since it's purely related to resource usage, and doesn't impact non-module-linking modules at all. cc bytecodealliance#2094
alexcrichton
added a commit
to alexcrichton/wasmtime
that referenced
this issue
Dec 2, 2020
This commit implements the final piece of the module linking proposal which is to flesh out the support for importing/exporting instances and modules. This ended up having a few changes: * Two more `PrimaryMap` instances are now stored in an `Instance`. The value for instances is `InstanceHandle` (pretty easy) and for modules it's `Box<dyn Any>` (less easy). * The custom host state for `InstanceHandle` for `wasmtime` is now `Arc<TypeTables` to be able to fully reconstruct an instance's types just from its instance. * Type matching for imports now has been updated to take instances/modules into account. One of the main downsides of this implementation is that type matching of imports is duplicated between wasmparser and wasmtime, leading to posssible bugs especially in the subtelties of module linking. I'm not sure how best to unify these two pieces of validation, however, and it may be more trouble than it's worth. cc bytecodealliance#2094
alexcrichton
added a commit
that referenced
this issue
Dec 2, 2020
This commit is intended to do almost everything necessary for processing the alias section of module linking. Most of this is internal refactoring, the highlights being: * Type contents are now stored separately from a `wasmtime_env::Module`. Given that modules can freely alias types and have them used all over the place, it seemed best to have one canonical location to type storage which everywhere else points to (with indices). A new `TypeTables` structure is produced during compilation which is shared amongst all member modules in a wasm blob. * Instantiation is heavily refactored to account for module linking. The main gotcha here is that imports are now listed as "initializers". We have a sort of pseudo-bytecode-interpreter which interprets the initialization of a module. This is more complicated than just matching imports at this point because in the module linking proposal the module, alias, import, and instance sections may all be interleaved. This means that imports aren't guaranteed to show up at the beginning of the address space for modules/instances. Otherwise most of the changes here largely fell out from these two design points. Aliases are recorded as initializers in this scheme. Copying around type information and/or just knowing type information during compilation is also pretty easy since everything is just a pointer into a `TypeTables` and we don't have to actually copy any types themselves. Lots of various refactorings were necessary to accomodate these changes. Tests are hoped to cover a breadth of functionality here, but not necessarily a depth. There's still one more piece of the module linking proposal missing which is exporting instances/modules, which will come in a future PR. It's also worth nothing that there's one large TODO which isn't implemented in this change that I plan on opening an issue for. With module linking when a set of modules comes back from compilation each modules has all the trampolines for the entire set of modules. This is quite a lot of duplicate trampolines across module-linking modules. We'll want to refactor this at some point to instead have only one set of trampolines per set of module linking modules and have them shared from there. I figured it was best to separate out this change, however, since it's purely related to resource usage, and doesn't impact non-module-linking modules at all. cc #2094
alexcrichton
added a commit
to alexcrichton/wasmtime
that referenced
this issue
Dec 2, 2020
This commit implements the final piece of the module linking proposal which is to flesh out the support for importing/exporting instances and modules. This ended up having a few changes: * Two more `PrimaryMap` instances are now stored in an `Instance`. The value for instances is `InstanceHandle` (pretty easy) and for modules it's `Box<dyn Any>` (less easy). * The custom host state for `InstanceHandle` for `wasmtime` is now `Arc<TypeTables` to be able to fully reconstruct an instance's types just from its instance. * Type matching for imports now has been updated to take instances/modules into account. One of the main downsides of this implementation is that type matching of imports is duplicated between wasmparser and wasmtime, leading to posssible bugs especially in the subtelties of module linking. I'm not sure how best to unify these two pieces of validation, however, and it may be more trouble than it's worth. cc bytecodealliance#2094
alexcrichton
added a commit
to alexcrichton/wasmtime
that referenced
this issue
Dec 3, 2020
This commit implements the final piece of the module linking proposal which is to flesh out the support for importing/exporting instances and modules. This ended up having a few changes: * Two more `PrimaryMap` instances are now stored in an `Instance`. The value for instances is `InstanceHandle` (pretty easy) and for modules it's `Box<dyn Any>` (less easy). * The custom host state for `InstanceHandle` for `wasmtime` is now `Arc<TypeTables` to be able to fully reconstruct an instance's types just from its instance. * Type matching for imports now has been updated to take instances/modules into account. One of the main downsides of this implementation is that type matching of imports is duplicated between wasmparser and wasmtime, leading to posssible bugs especially in the subtelties of module linking. I'm not sure how best to unify these two pieces of validation, however, and it may be more trouble than it's worth. cc bytecodealliance#2094
alexcrichton
added a commit
that referenced
this issue
Dec 3, 2020
* Implement imported/exported modules/instances This commit implements the final piece of the module linking proposal which is to flesh out the support for importing/exporting instances and modules. This ended up having a few changes: * Two more `PrimaryMap` instances are now stored in an `Instance`. The value for instances is `InstanceHandle` (pretty easy) and for modules it's `Box<dyn Any>` (less easy). * The custom host state for `InstanceHandle` for `wasmtime` is now `Arc<TypeTables` to be able to fully reconstruct an instance's types just from its instance. * Type matching for imports now has been updated to take instances/modules into account. One of the main downsides of this implementation is that type matching of imports is duplicated between wasmparser and wasmtime, leading to posssible bugs especially in the subtelties of module linking. I'm not sure how best to unify these two pieces of validation, however, and it may be more trouble than it's worth. cc #2094 * Update wat/wast/wasmparser * Review comments * Fix a bug in publish script to vendor the right witx Currently there's two witx binaries in our repository given the two wasi spec submodules, so this updates the publication script to vendor the right one.
alexcrichton
added a commit
to alexcrichton/wasmtime
that referenced
this issue
Dec 3, 2020
This commit does everything necessary (ideally) to support the module linking proposal in the C API. The changes here are: * New `wasm_{module,instance}type_t` types and accessors * New `wasm_{module,instance}_type` functions * Conversions between `wasm_extern_t` and `wasm_{instance,module}_t`, as well as `wasm_externtype_t` and the new types. * Addition of `WASM_EXTERN_{MODULE,INSTANCE}` constants * New `wasm_config_t` modifier to enable/disable module linking With these functions it should be possible to pass instances/modules to instances and also acquire them from exports. Altogether this should enable everything for module linking. An important point for this is that I've opted to add all these items under the `wasm_*` name prefix instead of `wasmtime_*`. I've done this since they're all following the idioms of existing APIs and while not standard the intention would be to standardize them (unlike many other Wasmtime-specific APIs). cc bytecodealliance#2094
alexcrichton
added a commit
to alexcrichton/wasmtime
that referenced
this issue
Dec 3, 2020
This commit does everything necessary (ideally) to support the module linking proposal in the C API. The changes here are: * New `wasm_{module,instance}type_t` types and accessors * New `wasm_{module,instance}_type` functions * Conversions between `wasm_extern_t` and `wasm_{instance,module}_t`, as well as `wasm_externtype_t` and the new types. * Addition of `WASM_EXTERN_{MODULE,INSTANCE}` constants * New `wasm_config_t` modifier to enable/disable module linking With these functions it should be possible to pass instances/modules to instances and also acquire them from exports. Altogether this should enable everything for module linking. An important point for this is that I've opted to add all these items under the `wasm_*` name prefix instead of `wasmtime_*`. I've done this since they're all following the idioms of existing APIs and while not standard the intention would be to standardize them (unlike many other Wasmtime-specific APIs). cc bytecodealliance#2094
alexcrichton
added a commit
that referenced
this issue
Dec 3, 2020
* Update the C API with module linking support This commit does everything necessary (ideally) to support the module linking proposal in the C API. The changes here are: * New `wasm_{module,instance}type_t` types and accessors * New `wasm_{module,instance}_type` functions * Conversions between `wasm_extern_t` and `wasm_{instance,module}_t`, as well as `wasm_externtype_t` and the new types. * Addition of `WASM_EXTERN_{MODULE,INSTANCE}` constants * New `wasm_config_t` modifier to enable/disable module linking With these functions it should be possible to pass instances/modules to instances and also acquire them from exports. Altogether this should enable everything for module linking. An important point for this is that I've opted to add all these items under the `wasm_*` name prefix instead of `wasmtime_*`. I've done this since they're all following the idioms of existing APIs and while not standard the intention would be to standardize them (unlike many other Wasmtime-specific APIs). cc #2094 * Appease doxygen
Ok this is all done now so I'm going to close this. I'm still wading through fuzz bugs at this point but the bulk of it is all there! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I plan to use this as a tracking issue for the module linking proposal in Wasmtime. I'll be updating this description over time as I find time and as work is done:
Implementation Notes
Some miscenalleous notes on the implementation and how this is being implemented:
wasmtime::Module
will internally have a list of modules to select from.Resolver
for the actual imports, as well as the "surrounding environment" which is probably "the list of all other modules that came from the original wasm file".Open questions:
wasm2obj
do for multi-module wasm files?wasmtime-jit
be for imported instances and imported modules?The text was updated successfully, but these errors were encountered: