Serialization and encoding/decoding #4769
Comments
IGI-111
added
compiler
|
After some investigation, it looks like the current way our encoding works is different for input and output. Input values are directly usable and pointer values are populated with correct memory offsets so they can be directly used. Output values are fully dereferenced and contain no indirection, which means it's hard/impossible to nest them. Any dynamically sized return value will have to be named last and can't have siblings because it will consume the entire rest of the buffer when decoding. Note that these two encodings are not compatible with each other, which we'll definitely want to solve. What we should do instead is similar to Solidity's encoding scheme: split the encoded buffer into two regions, one for inline bytes and the other for dynamic data, and have all pointers values be encoded inline with pointer values that are offsets of the beginning of the data region. Then, in Sway, for both inputs and outputs, we should parse these into proper pointers for inputs, and generate proper data section offsets for outputs. |
|
There is a practical issue here in that the test frameworks for sway and fuels-rs are both dependent on each other, and changing this serialization will require a breaking change of both. We'll have to figure out how to choreograph this. Likely by bypassing one of the test suites after we've manually confirmed the first change works on the updated version of the other repository. |
|
blocked by: #4929 |
## Description This PR changes the output of `__log` to encoded values (see #4769). A quick example of what that means is ```sway struct S { a: u64, b: u32, c: u16, d: u8, e: Vec<u64>, f: str, g: u256 } let mut e = Vec::new(); e.push(1); e.push(2); e.push(3); __log(S{ a: 1, b: 2, c: 3, d: 4, e, f: "sway", g: u256::max() }); ``` will output ``` [0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 2, 0, 3, 4, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 4, 115, 119, 97, 121, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255] ^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^ ^^^^ ^ ^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^ ^^^ ^^^ ^^ ^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ s.a s.b s.c s.d s.e.len() s.e[0] s.e[1] s.e[2] s.f.len() s.f[0] s.f[1] s.f[2] s.f[3] s.g ``` This works in two steps: 1 - `__log(s)` is desugared into `__log(encode(s))`; 2 - call `encode`. Its impl can be found at https://github.com/FuelLabs/sway/pull/5306/files#diff-ee5cebc963e841e8af05f3986de17dd266ee6e9b49dbe089a5eb64764f3b802eR307 It simply creates an append-only buffer and call `abi_encode` from a special trait named `AbiEncode`. To be encodable, a type must implement `AbiEncode`. In the example above, `S` is auto-implemented by the compiler because all its fields are `AbiEncode`. But we can also have custom impl like `Vec` here: https://github.com/FuelLabs/sway/pull/5306/files#diff-b5d9688741fea479477f26ca44cd1d1ecbd2f003f3875292abb23df7fad85c58 All this is behind a compiler flag: ``` > forc build -h Compile the current or target project USAGE: forc build [OPTIONS] OPTIONS: ... --experimental-new-encoding Experimental flags for the "new encoding" feature ``` The same flag is available for the `e2e` tests: ``` > cargo r -p test --release -- should_pass/language/logging --verbose --experimental-new-encoding ``` ## Limitations 1 - Now that __log demands a `fn` called `encode`, when something is compiled with `implicit-std: false`, the intrinsic function `__log` does not work out-of-box anymore. A function called `encode` must "visible" and the worst part is that it needs to be functional to log anything. 2 - Arrays, string arrays and tuples will have limited implementations. Currently up to five items. ## Checklist - [x] I have linked to any relevant issues. - [x] I have commented my code, particularly in hard-to-understand areas. - [x] I have updated the documentation where relevant (API docs, the reference, and the Sway book). - [x] I have added tests that prove my fix is effective or that my feature works. - [x] I have added (or requested a maintainer to add) the necessary `Breaking*` or `New Feature` labels where relevant. - [x] I have done my best to ensure that my PR adheres to [the Fuel Labs Code Review Standards](https://github.com/FuelLabs/rfcs/blob/master/text/code-standards/external-contributors.md). - [x] I have requested a review from the relevant team or maintainers. --------- Co-authored-by: Joshua Batty <joshpbatty@gmail.com>
kamyar-tm
added
the
epic
Binary encoding of arbitrary sway values is a necessary feature, notably to return values/pass values to contracts in a predictable, portable fashion.
This is tricky and solidity has had to go through multiple versions of it but we can borrow from their encoding scheme.
This should take versioning into account, as we'll likely want to make alterations to this scheme in the future.
We should also consider options for packing the data (such as Solidity's strict mode).
We should also consider the ergonomics of how to use it in the language, a trait based approach seems natural but we don't have derive macroes so having people implement a trait could prove tedious. We may want to rely on intrinsics to make the interface nicer.
The text was updated successfully, but these errors were encountered: