Feasibility of Golang for Substrate Runtimes - Jan 2021

[dutterbutter]

Feasibility of Golang for Substrate Runtimes - Jan 2021

Go has been the defacto language of blockchain and distributed systems for much of the past decade. As a result there is a vast quantity of implementations and expertise contained within the ecosystem.

One of the major benefits of the Substrate framework is that it supports runtimes (state transition logic) written in WASM. Currently only Rust is officially supported by Parity as a language for authoring runtimes but in theory any WASM supported language could be used.

This documented investigates the suitability of Go for this purpose as of Jan 2021 and outlines any blockers and work required.

State of Go->WASM

For compiling Go to WASM there are two main paths that can be taken.

The first is to use the regular Go compiler. As of version Go 1.11, WASM has been added as an experimental compilation target. The resulting WASM binary contains a full runtime as well as the compiled code logic. From the 1.11 release notes:

Go programs currently compile to one WebAssembly module that includes the Go runtime for goroutine scheduling, garbage collection, maps, etc. As a result, the resulting size is at minimum around 2 MB, or 500 KB compressed.

The second approach is to use TinyGo. This is an alternative LLVM based compiler for the language that aims to target microcontrollers as well as WASM. It supports a subset of the standard language features but aims to be able to compile most most code without modification.

Package Compatibility - Go vs TinyGo

In order to verify what can be compiled to WASM in both the Go and TinyGo a test binary was built that imports and calls basic functionality in the following:

• Calculating a SHA2 hash
• RLP encoding (from Geth)
• Schnorr signing and verifying
• SCALE decoding (from Gossamer)

If it successfully compiled the binary size was recorded. Code can be found here

Package	Go	TinyGo
crypto/sha256	2.2MB	249.6kB
go-ethereum/rlp	3.5MB	❌
ChainSafe/go-schnorrkel	2.7MB	❌
ChainSafe/gossamer/lib/scale	2.9MB	❌

In the case of both RLP and SCALE the TinyGo build failed due to lacking features in "reflect" and for Schnorr signing it reported an unsupported instruction.

Given the above it appears at this stage that TinyGo has not yet reached compatibility with much of the ecosystem. Go produces large WASM modules however this is less of an issue for runtimes as it is for browser modules or smart contracts.

Requirements for Substrate Runtimes

In a node a runtime a WASM module is executed inside of an interpreter (WASMI?) and invoked whenever a new block is processed or created in order to update the state. The Runtime WASM module must export particular functionality to the node in order for this to happen. In Substrate this is called the Runtime API.

The requirements of the API are particular to the node however there is a core API which must be implemented by all. The Core API is very minimal and only requires the module to export:

fn version() -> RuntimeVersion;
fn execute_block(block: Block);
fn initialize_block(header: &<Block as BlockT>::Header);

In Substrate FRAME, the runtime API functions are exposed using the impl_runtime_apis! macro. This produces an implementation similar to:

#[no_mangle]
pub unsafe fn fn_name(input_data: *mut u8, input_len: usize) -> u64 { }

which results in an exported callable function in the resulting WASM module.

In TinyGo, a similar result can be reached by declaring:

//go:export fn_name
func fn_name(input_data, input_len int) int64 { }

Currently it is not possible to build a WASM module in Go that exports functions. It only supports calling from Javascript. See golang/go#42372.

When compiled to WASM this will expose a function with the given name that takes a pointer and size to the input data and returns a u64 which may be a pointer to some output data. Recall this is because the only types supported by WASM are 32 and 64 bit integers or floats.

The node can also expose native functions callable from the runtime (the opposite to above). This is called the Runtime Interface. Details on how the function calls out are implemented in WASM can be researched in detail at a later date.

Required to Progress

• A working way to author WASM modules with function exports in Go
• OR TinyGo support for more of the ecosystem

Substrate Specifics

To reach a basic working runtime requires:

• Compatible memory [de]serialization to allow function parameters and return values to be passed across the WASM boundary.
• Implementation of the Core Runtime API

A developer friendly implementation requires:

• Some way to wrap functions to abstract the WASM memory handling (e.g. impl_runtime_apis! macro in Rust)
• Reimplementation of core types from Substrate FRAME with identical memory serialization for compatibility

Links

• https://github.com/golang/go/wiki/WebAssembly
• https://tinygo.org/
  • https://github.com/tinygo-org/tinygo/tree/master/src/examples/wasm
  • https://tinygo.org/webassembly/webassembly/
• https://github.com/golang/proposal/blob/master/design/42372-wasmexport.md

[noot]

issue I opened a while back: tinygo-org/tinygo#1824 unknown at this point if wasm compiled from Go can access the module memory

[EclesioMeloJunior]

Added my 2 cents to that issue!

[Noc2]

I think we would be quite interested in supporting this. Just for reference, you might want to take a look at https://github.com/LimeChain/subsembly. They use AssemblyScript for substrate runtimes.