NEP-481: Synchronous wasm submodules

README.md

@@ -29,6 +29,7 @@ Changes to the protocol specification and standards are called NEAR Enhancement
 | [0399](https://github.com/near/NEPs/blob/master/neps/nep-0399.md) | Flat Storage                             | @Longarithm @mzhangmzz                      | Review |
 | [0448](https://github.com/near/NEPs/blob/master/neps/nep-0448.md) | Zero-balance Accounts                    | @bowenwang1996                              | Final  |
 | [0455](https://github.com/near/NEPs/blob/master/neps/nep-0455.md) | Parameter Compute Costs                  | @akashin @jakmeier                          | Final  |
+| [0481](https://github.com/near/NEPs/blob/master/neps/nep-0481.md) | Synchronous wasm submodules                  | @mooori @birchmd                          | New  |
 
 ## Specification
 

neps/nep-0481.md

@@ -0,0 +1,145 @@
+---
+NEP: 481
+Title: Synchronous wasm submodules
+Authors: Moritz Zielke <moritz.zielke@aurora.dev>, Michael Birch <michael.birch@aurora.dev>
+Status: New
+DiscussionsTo: https://github.com/nearprotocol/neps/pull/0481
+Type: Protocol
+Version: 1.0.0
+Created: 2023-05-12
+LastUpdated: 2023-05-12
+---
+
+# Synchronous wasm submodules
+
+## Summary
+
+This NEP proposes to enable the synchronous execution of wasm submodules in the NEAR runtime. Submodules are deployed under a NEAR account (the _parent_). A submodule may contain arbitrary wasm instructions and has access to submodule host functions. The execution of a submodule is triggered by the parent contract invoking a host function to start the submodule. The submodule may suspend itself by yielding back to the parent, which may subsequently resume the submodule. Upon yield and resume, data can be exchanged bidirectionally between parent and submodule.
+
+## Motivation
+
+Currently all interactions between NEAR contracts are asynchronous, which prevents use cases that require synchronous interactions. An example of such a use case is Aurora.
+
+Aurora is a smart contract on NEAR which emulates an Ethereum ecosystem. To execute a function call on an Ethereum contract, it loads the contract and its state from storage, spins up an EVM interpreter to execute the contract’s EVM bytecode and persists state changes. Interpreting EVM bytecode inside a NEAR transaction is inefficient in terms of gas usage.
+
+Given a [compiler](https://github.com/aurora-is-near/evm2near) to convert EVM bytecode into wasm bytecode targeted at NEAR, Aurora can [reduce its gas consumption](https://github.com/aurora-is-near/aurora-engine/pull/463) significantly. Instead of spinning up an EVM interpreter and interpreting a contract’s EVM bytecode it would invoke a submodule whose wasm is equivalent that EVM bytecode.
+
+Significantly increasing the efficiency of Aurora can be benefical for the NEAR ecosystem in general. Beyond that, the modifications of NEAR that are required for synchronous wasm submodules can be a building block for [account extensions](https://gov.near.org/t/proposal-account-extensions-contract-namespaces/34227).
+
+### wasm submodules and account extensions
+
+The proposed wasm submodules can be interpreted as account extensions with tight restrictions. A wasm submodule is an account extension that can be executed only by the parent contract itself. It can access only submodule host functions which  allow it to yield back to the parent and exchange data with it. Host functions available to regular contracts cannot be invoked by the submodule. A submodule cannot read or write the parent’s storage and has no storage associated with itself.
+
+Due to its complexity, account extension functionality might be implemented in multiple stages. Synchronous wasm submodules could be an initial stage, which progresses towards account extensions as the restrictions mentioned above are lifted.
+
+## Specification
+
+### Deploying and storing submodules
+
+Submodules are deployed to accounts via the `DeploySubmodule` action. Multiple submodules can be deployed to a single account whereby each is stored under a separate key.
+
+To prevent contracts from (accidentaly) modifying submodules, they are persisted apart from the accounts storage. The data making up submodules can only be written via `DeploySubmodule` and `RemoveSubmodule` actions, i.e. not via the host functions `storage_write` or `storage_remove`.
+
+### Starting and resuming host functions
+
+A contract can start and resume submodules deployed to its account via host functions. A submodule must be started first before it can be resumed.
+
+### Data exchange between parent and submodule
+
+When resuming a submodule, the parent can pass data to the submodule. When yielding to the parent, a submodule can pass data back. Data exchanged between parent and submodule are bytes.
+
+### Submodule host functions
+
+The submodule can import host functions to read data received from the parent, yield back to the parent and set a return value on termination. A `gas` host function must be available for gas accounting and to meet the [requirements](https://github.com/near/nearcore/blob/83b1f80313ec982a6e16a1941d07701e46b7fc35/runtime/near-vm-runner/src/instrument/gas/mod.rs#L396-L402) of nearcore wasm instrumentation.
+
+### Trustless
+
+It is possible to deploy submodules from untrusted sources without introducing security vulnerabilities for the parent. The specification proposed here ensures this by limiting the host functions available to submodules. The interface of passing bytes between parent and submodules allows each contract to implement submodule permissions suited for its use case.
+
+## Reference implementation
+
+A proof of concept (PoC) implementation is available at [this branch](https://github.com/birchmd/nearcore/tree/sync-wasm-poc). The following sections refer to this PoC.
+
+### Status
+
+The PoC implementation allows writing submodules in Rust and executing them synchronously. [Here](https://github.com/birchmd/nearcore/blob/67ce064e5a692c9af10f8b7ae23606218d78a244/runtime/near-test-contracts/test-submodule-rs/src/lib.rs) is a submodule implemented in Rust which calls host functions and interacts with the parent contract. It can be [compiled to wasm](https://github.com/birchmd/nearcore/blob/67ce064e5a692c9af10f8b7ae23606218d78a244/runtime/near-test-contracts/build.rs#L31) like a regular contract. In [this test](https://github.com/birchmd/nearcore/blob/67ce064e5a692c9af10f8b7ae23606218d78a244/integration-tests/src/tests/runtime/submodule.rs#L318-L346), the submodule is deployed and then executed via the parent contract by calling [execute_submodule_rs](https://github.com/birchmd/nearcore/blob/67ce064e5a692c9af10f8b7ae23606218d78a244/runtime/near-test-contracts/test-contract-rs/src/lib.rs#L1416-L1479).
+
+### Implementation details
+
+The wasm bytecode of submodules is subject to the same instrumentation as the wasm bytecode of regular contracts. Control flow passing is handled on the host using [corosonsei coroutines](https://docs.rs/corosensei/latest/corosensei/). The host function to start a submodule spins up a virtual machine in which the submodule’s bytecode is executed. This virtual machine is contained in a coroutine and when the submodule yields back to the parent, the host suspends the coroutine which interrupts execution of the submodule. Resuming the execution of the submodule is accomplished by resuming the coroutine.
+
+The gist of the PoC is implemented in [wasmer2_runner.rs](https://github.com/birchmd/nearcore/blob/6cac4924997f658ec8b91a58e1513564402b7f1b/runtime/near-vm-runner/src/wasmer2_runner.rs).
+
+### Outstanding TODOs
+
+The current implementation is in a PoC state. The most important outstanding tasks are:
+
+- Gas accounting: Gas cost of operations that handle submodules need to be estimated and accounted for in the newly added host functions.
+- Submodule storage: Submodule bytecode has to be stored such that it is writable only via actions `DeploySubmodule` and `RemoveSubmodule`.
+- Removing submodules: Support for the action `RemoveSubmodule` has to be added.
+- Error handling:
+	- Not all errors related to submodules that may occur inside the host are handled properly (some errors are `unwrap()`’ed in the PoC).
+	- Traps occuring during the execution of submodule bytecode need to be handled.
+
+## Alternatives
+Possible alternatives for the synchronous execution of wasm bytecode are:
+
+### An `eval` host functions
+
+An `eval` host function that executes bytecode which it received as input parameter.
+
+Pros:
+
+- Might allow the execution of bytecode without previous deployment, which could reduce the complexity of some use cases.
+
+Cons:
+- Coroutines are meant to run concurrently and be resumable, which are features required for supporting the concurrent execution of submodules. Supporting this functionality with an `eval` function might be more involved.
+- Handling traps in submodules is likely to be more challenging when the submodule is executed in the same virtual machine as the parent.
+- In some programming languages similar `eval` functions have a notoriously bad reputation, see for instance [Python](https://stackoverflow.com/questions/1832940/why-is-using-eval-a-bad-practice/1832957#1832957) and [JavaScript](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/eval#never_use_eval!).
+
+### Account extensions
+
+_Note that the discussion regarding the scope and specification of account extensions is still ongoing. As of 2023-05-10, a NEP for account extensions does not yet exist._
+
+The synchronous execution of wasm bytecode should, ultimately, be possible with account extensions. In that case, setting appropriate submodule permissions should serve the use cases that would be enabled by this proposal. However, account extensions are much broader in scope and accordingly their implementation is expected to be significantly more complex. Instead of being an alternative, synchronous wasm submodules might rather be a building block for account extensions.
+
+## Drawbacks
+
+### Passing data by exchanging bytes
+
+Limiting the interface between a parent and its submodules to passing bytes introduces overhead in cases that require interaction. For instance, instead of directly persisting data in storage a submodule has to serialize data and send it back to the parent. The parent then needs to deserialize the data, write it to storage and resume the submodule. Besides requiring extra logic this pattern also increases the number of host function calls.
+
+That overhead could be reduced by extending the interface, for instance by making more host functions available to submodules. The trade-off with giving submodules direct access to more host functions is complexity in permissions.
+
+For example if submodules were allowed their own state and had their own versions of `storage_read` and `storage_write` then the parent should be able to set limits on the size of this storage to prevent submodules from incurring too much storage staking costs. Similarly, if submodules could create promises then the parent would likely want to limit which functions could be called and how much gas could be attached (similar to how function call access keys work for accounts).
+
+## Security Implications
+
+## Future possibilities
+
+## Changelog
+
+### 1.0.0 - Initial Version
+
+- Created 2023-05-12
+
+#### Benefits
+
+[Placeholder for Subject Matter Experts review for this version:]
+
+- Benefit 1
+- Benefit 2
+
+#### Concerns
+
+[Template for Subject Matter Experts review for this version:]
+| # | Concern | Resolution | Status |
+| - | - | - | - |  
+| 1 | | | |
+| 2 | | | |
+
+## Copyright
+
+[copyright]: #copyright
+
+Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).