Implementing Clarity Extensions for 2.1

[kantai]

In Stacks 2.1, we want to implement new native functionality for Clarity. However, implementing this new functionality requires new reserved words. This is problematic, because adding reserved words is not a backwards-compatible change. There’s potentially two ways to deal with this:

1. Adding a pragma/code marker that indicates that a given contract is “Clarity Version 2”
2. Using contract-call interposition and “extension contracts” to avoid the use of reserved words entirely

Advantages / Disadvantages

The pragma approach is more simple for developers and provides greater extensibility: it could be used in the future to add new types, etc. The disadvantage of the pragma approach is that it will require the interpreter and analysis passes to operate “context-aware”, which is a greater maintenance burden long term.

Contract-call interposition and “extension contracts” are more limited: they cannot introduce new types or new “specialy-typed functions”, rather they must have the same semantics as contract-calls, but they can be implemented without the VM or the analysis passes being context-aware.

To me, it seems like the "pragma approach" would be better: it provides a better experience for developers and is more extensible for the future. The downside is it is probably harder to implement and maybe harder to test, though both approaches require the same kind of testing described below. I'd be hard pressed to estimate how difficult the pragma approach would be to implement.

Implementing via Contract-Call Interposition

In order to implement Clarity extensions via a contract-call, a few things must happen.

1. If a Stacks block is the “first” Stacks block in its fork following the epoch switch, the extension contracts must be published.
2. Once a Stacks block is executed after (1), contract-call interposition will be instantiated. This is a check in the implementation of contract-call that checks whether or not the called contract is an extension contract. If it is, is checks whether the function name is an extension function, and then executes the corresponding “native” implementation, rather than the normal contract-call implementation.
3. Once a Stacks block is executed after (1), contract-call interposition must also be done in the analysis passes. The normal analysis pass behavior for contract-call is just look up the corresponding contract and the function’s typing. However,

Desugaring Pass

One nice-to-have feature would make this less hostile of an interface for developers: extension desugaring: essentially, a contract author could write

;; uses 2.1

(new-function arg1 arg2)

In their contracts, and a desugaring pass would rewrite the contract to

;; uses 2.1

(contract-call? SP000000000000000000002Q6VF78.extensions new-function arg1 arg2)

Implementing via Context-Aware VM

In order to implement these extensions via a context-aware VM and analysis passes (i.e., option 1 above), the ContractContext struct in the VM would need to track the current Clarity version. When a contract-call occurs to a contract, the Clarity version would switch to the called contract’s version.

The analysis passes would also need to be context-aware, however, this is slightly easier than the runtime contexts: the analysis passes do not “cross” contract boundaries: they simply fetch the already computed analysis for a given contract (which would never change).

Testing Clarity Extensions

All new Clarity extensions must be tested across the 2.0/2.1 epoch boundary.

This means that for a new function:

1. A 2.0 contract which defines a local function named with that same function name should continue to work exactly the same.
2. An attempt to publish a 2.0 contract which uses the new function should fail with an analysis error.
3. A 2.1 contract published with the new function should work.
4. A 2.1 contract which calls a 2.0 contract should work:
   1. The 2.1 contract should be able to use the new function before calling the 2.0 contract
   2. The 2.0 contract should be able to call its own function with the same name
   3. The 2.1 contract should be able to use the new function after calling the 2.0 contract (whether or not the 2.0 contract returned an ok/err)
5. A 2.0 contract which calls a 2.1 contract (via trait) should work
   1. The 2.0 contract should be able to call its own function with the same name
   2. The 2.1 contract should be able to use the new function
   3. The 2.0 contract should be able to call its own function with the same name after calling the 2.1 contract
6. (at-block …) expressions that cross the 2.0/2.1 boundary should work
   1. If implemented via “extension contract”, an (at-block ..) expression that tries to use one of the new functions in the past should yield an error.
   2. If implemented via “pragma”, the (at-block …) expression should still be able to use one of the new functions

[psq]

by desugaring to a contract-call, wouldn't this require all extensions to have a return type of (response ... ...)? Maybe the desugaring could be a bit more flexible to allow including unwrapping when it makes sense?

[jcnelson]

That's correct, but this would only be for extension contract functions that are declared as define-public functions (if they were define-read-only then they could return whatever they wanted).

[jcnelson]

The disadvantage of the pragma approach is that it will require the interpreter and analysis passes to operate “context-aware”, which is a greater maintenance burden long term.

Are there any non-contrived programs or data structures that we need to write in Clarity that cannot be built with the extension contract method? I can't think of any new types we need to introduce.

The only "specially-typed function" I can think of that we would really like to have is a slice operation on buffers, but I'm not sure that we have to implement it. If we instead implemented slice-1, slice-2, slice-4, slice-8, slice-16, etc., where slice-$i returned (buff $i) (we'd go up to slice-524288), then that might be enough.

These could all be implemented as an extension contract. Callers would just use the smallest slice-$i that suited their purposes. I've already written a generic slice function that uses slice-$i implementations simply by greedily reading the largest possible slice from the remaining range; we could put that into an extension contract as well, and give it a cost function equal to the resulting slice length.

The reason I bring this up is because I suspect the pragma approach is going to have lots of tiny gotchas in it. You'll basically be peppering the whole VM with a bunch of if epoch_id == StacksEpochId::Epoch21 { /* do 2.1 stuff */ } else { /* do 2.0 stuff */ } code blocks, and that can quickly create a very large set of possible test cases. If we can avoid this, or convince ourselves that we can gate the parser, analysis passes, and VM execution accordingly, then I might be onboard with this.

By contrast, the code I've written that replaces function bodies with native code was surprisingly simple, once I was familiarized enough with the codebase. The only gating logic that has to occur at all is the instantiation of the extension contracts; once they exist, the function eval() code simply looks to see if the given function has a native implementation, and calls that instead. Trivially, the extension contracts will never be called in epoch 2.0, because they don't exist.

[jcnelson]

Two things:

• We can implement shadowing through a pragma. In 2.0, shadowing reserved words in 2.1 is permitted by default; in 2.1, it is forbidden by default, but may be re-enabled through a pragma.

• Using pragmas instead of extension contracts not only obviates the need for replacing function bodies with native implementations, but also front-loads a bunch of work that would need to be done to fix a catastrophic failure in the Clarity VM.

[kantai]

Relevant at-block example

(at-block <foo>
    (slice buff 10))

[jcnelson]

If we go the route of using Clarity versioning instead of extension contracts, then what does it mean for extensions to .pox? Are these going to be native keywords, or do we still need to write new .pox methods and implement contract-call interception logic?

[kantai]

The plan for pox doesn't involve extensions in any event (except, maybe, for fetching something like get-account-lock-details, which just returns the lock details for an account entry, but that's not PoX specific). Rather, we're just launching a pox-2 contract, which implements everything for the upgraded PoX.

[jcnelson]

Just an update on this: the decision we've landed on is to implement new Clarity keywords natively, and implement a Clarity contract versioning scheme and pragma keyword whereby new keywords will only be available to versions of Clarity in which they are supported. Moreover, the Clarity VM version will be fixed in the global context and unchangeable throughout the contract or contract-call execution -- calls to at-block that call into code in older versions of Clarity will run with the same Clarity VM rules as the outermost execution context.

Extension contracts will not be used, since they are not needed.

[psq]

sounds like a good plan!