diff --git a/eip-0008.md b/eip-0008.md new file mode 100644 index 00000000..3fbc5d50 --- /dev/null +++ b/eip-0008.md @@ -0,0 +1,438 @@ +# New Transaction Validation with Just-In-Time Costing + +* Author: aslesarenko +* Status: Proposed +* Created: 10-Mar-2020 +* License: CC0 +* Forking: soft-fork needed + +## Contents +- [Background And Motivation](#background-and-motivation) +- [Ergo Transaction Validation With JIT Costing](#ergo-transaction-validation-with-jit-costing) + - [Transaction Verification](#transaction-verification) + - [Sending Transaction](#sending-transaction) + - [Maximum Transaction Cost](#maximum-transaction-cost) + - [Context-Independent Scripts](#context-independent-scripts) + - [New Costing of ErgoTree](#new-costing-of-ergotree) +- [References](#references) +- [Appendix A: AOT vs JIT Costing](#appendix-a-aot-vs-jit-costing) + - [AOT Costing](#aot-costing) + - [JIT Costing](#jit-costing) +- [Appendix B: Benefits of JIT costing](#appendix-b-benefits-of-jit-costing) + +## Background And Motivation + +Every unspent box is protected by a contract script, which is stored in the box itself as +serialized [ErgoTree](https://ergoplatform.org/docs/ErgoTree.pdf). The script should be +evaluated as part of transaction validation to verify box's spending conditions. + +Script verification takes some time and system resources on every network node and thus +both the execution time and the resources need to be put under control. This is known as +cost estimation (or "costing"). + +Costing is necessary to prevent long delays in verification of new blocks by network +nodes. Both over-sized and over-expensive scripts may be exploited to attack the network +by exhausting verifiers' resources and slowing down network transaction throughput. + +There are two general approaches to do cost control: +1) Ahead-Of-Time Costing (AOTC) - estimation of the execution costs in which the cost of +the script is approximated before and without the actual script execution. +2) Just-In-Time Costing (JITC) - accumulation of the execution cost during +the actual contract execution. + +This EIP describes a hybrid JIT+AOT approach for costing of Ergo contracts verification. + +Note, AOTC and JITC approaches are compared in [Appendix +A](#appendix-a-aot-vs-jit-costing) and the potential benefits of JITC are described in +[Appendix B](#appendix-b-benefits-of-jit-costing). + +## Ergo Transaction Validation With JIT Costing + +This EIP describes a hybrid JIT+AOT approach for costing of Ergo contracts verification, +which uses JIT costing for script reduction to sigma-protocol proposition (aka SigmaProp) +and then uses AOT costing to estimate cost of computationally very expensive cryptographic +operations to verify the SigmaProp proposition. + +The hybrid approach allows preventing execution of scripts beyond allotted limits. + +The implementation doesn't require changes in the serialization format and +interpretation of transactions in the block. Only cost verification is changed with this +proposal, which however require activation via soft-fork. Thus, the current version of Ergo +Protocol v2 must be increased to v3. + +The proposed change is conservative so that the new Ergo Protocol v2 is backward +compatible: +1) All existing applications (like exchanges, ErgoMix, DEX) will not require any changes. +2) The new protocol doesn't incur additional overhead and instead reduce it. + +The cost calculation of ErgoTrees is implemented as it is defined in [the next +section](#new-costing-of-ergotree). + +We first describe how transactions of a block are validated. This _on-chain_ validation is +part of the consensus protocol. Transaction validation is a deterministic procedure which +should be the same for all nodes of the network. + +Next, we describe the steps which should be done _off-chain_ by applications in order to +create, sign and send new transactions. + +### Transaction Verification + +The verification is performed by _verifier_ nodes for each new transaction `tx` either +selected from the pool or contained in a new block candidate broadcast over the network. + +1) A verifier determines the `costLimit` parameter of the transaction which should be less +than `MaxTxCost`. The limit is computed as `costLimit = tx.feeOut.value / +tx.feeOut.R4[Long]` if the register is not empty, otherwise `costLimit = MaxTxCost`. +The value in the `R4` register is the NERG price of a unit of cost (`costPrice`). +The `MaxTxCost` parameter is described in the [section below](#maximum-transaction-cost). + +2) The transaction `tx` can be added to the block candidate if the currently accrued block +cost `blk.accCost` plus `costLimit` is less than `MaxBlockCost`. Validation of the +block fails immediately after the current `blk.accCost` exceeds `MaxBlockCost`. + +3) The verifier validates `tx` performing all the checks of the current Ergo Protocol (v1 at +the time of writing) except verification of scripts and proofs. If the transaction doesn't +pass these checks it is _malformed_ and the verifier rejects it. + +4) All operations performed by the verifier have an associated costs. These costs are +accumulated to the `tx.accCost` parameter and shouldn't exceed `costLimit`. The +`tx.accCost <= costLimit` check is performed every time `tx.accCost` is increased. When +the limit is exceeded the verification is aborted and `tx` is rejected as +_over-limit_ transaction. + +5) The initial checks of transaction validity have a cost for verifiers, since verifiers +need to spend computation resources. The cost is accrued to the total transaction cost +`tx.accCost`. If the initial checks pass the transaction is _well-formed_ otherwise the +transaction is rejected. + +6) The well-formed transaction is further validated by executing ErgoTrees for all inputs +and verifying the corresponding proofs. The costs of the ErgoTree execution and the +subsequent proofs verification are [computed](#new-costing-of-ergotree) as part of script +evaluation and accrued to the `tx.accCost`. When the validation proceeds from one input to +another the values of `tx.accCost` and `costLimit` are passed to the ErgoTree interpreter +as parameters, so that the validation of the transaction fails immediately after the +current `tx.accCost` exceeds `costLimit`. When it happens `tx` is rejected as +_over-limit_ transaction. + +7) If the transaction is successfully verified, the actual cost value +`tx.accCost` is accrued to `blk.accCost`. If the transaction is rejected then: +- If the verification is part of a new block candidate assembly the `tx` is dropped. +- If the verification is part of a block verification the whole block is also rejected. + +### Sending Transaction + +The following steps are performed by Ergo application _off-chain_. + +1) A transaction sender composes a new transaction `tx` and performs signing. +Transaction signing involves: + 1) Execution of the scripts for all the input boxes. + 2) Execution of sigma protocols to generate proofs. + + The above two steps are performed using the context of the blockchain at the time when + `tx` is expected to be added to the block (often the time of transaction signing). In + general, the current blockchain height, PreHeader and the last 10 block headers can be + used in the scripts which may influence the result of the scripts. + +2) The byproduct of the signing is that `tx.accCost` is computed which is the estimated +cost of executing all the the `tx` scripts in the given blockchain context. If scripts +execution doesn't depend on the current blockchain parameters (like `HEIGHT`, `headers` +etc.) then `tx.accCost` deterministically predicts execution cost in _any_ context which +has additional benefits (see [context-independent scripts](#context-independent-scripts)). + +3) When the transaction has a script which _depends_ on the context then the transaction +sender can _optionally_ specify the `costPrice` in the `tx.feeOut.R4` register, which will +be used to compute the `costLimit` of the transaction. + + The `costLimit` parameter is used by the `tx` verifier to reject over-limit + transactions and should be greater or equal to `tx.accCost` computed during `tx` + verification. + + If the sender chooses to specify `tx.feeOut.R4` he/she also need to put at least + `feeLimit = costLimit * costPrice` NERGs (NanoERGs) in the miner's `feeOut` box. + +4) When `tx` has predictable `tx.accCost` cost (i.e. all input scripts are +context-independent) then `tx` verification will never exceed `tx.accCost`, and thus +setting `feeOut.value = tx.accCost * costPrice` guarantees the transaction is not rejected +as over-limit. + +5) When `tx.accCost` is not predictable, which may happen when `tx` has at least one +context-dependent script (i.e. at least one script have conditions on `HEIGHT`, +`CONTEXT.headers`, etc), then the sender can either 1) choose values for `feeOut.value` +and `costPrice` based on estimation, or 2) choose a value for `feeOut.value` and use +`MaxTxCost` for `costPrice` (see [details](#maximum-transaction-cost)) + +### Maximum Transaction Cost + +It is [observed in [2]](#references) that when a script execution requires nontrivial +computation effort, practical attacks exist which either 1) waste miners’ computational +resources or 2) lead miners to accept incorrect script results. This _verifier's dilemma_ +drives rational miners to be well-incentivized to accept unvalidated blockchains. +The solution proposed (see [2], Lemma1) incentivizes correct execution of scripts by rational miners +under the condition that miners only accept to verify the transactions that have cost limit +bounded by `MaxTxCost = epsilon * MaxBlockCost / N` where `MaxBlockCost` is the limit on block +verification costs, `N` is the number of transactions in a block and `epsilon` is a +parameter representing the computational advantage of skipping verification (or skipping +probability). + +In the case of Ergo we fix `MaxTxCost` (which can be increased via voting) and then +`epsilon = MaxTxCost * N / MaxBlockCost` become dependent on the block. + +JITC verification with the fixed `MaxTxCost` satisfy _epsilon-computer_ requirements +of [2] and therefore rational miners incentivized to execute scripts and verify transactions. + +Thus motivated `MaxTxCost` can be used as default transaction limit when `tx.feeOut.R4` is +not specified by the sender. This is important for backward compatibility with existing +applications and in cases when estimation of `costLimit` is not possible (e.g. for +context-dependent scripts). + +Miners' reward per unit of cost is `tx.feeOut.value / costLimit` in a worst case, and +rational miners are incentivized to order pool transactions by this reward ratio. +Senders thus incentivized to estimate minimum possible `costLimit` and put it into +`tx.feeOut.R4`. + +### Context-Independent Scripts + +There is a large class of scripts which execution doesn't depend on the blockchain context +parameters (like `HEIGHT`, last `headers` etc.) (see references [[1]](#references)). +For such _context-independent_ scripts the execution depends only on `tx` content (which +is immutable) and on nothing else. + +The "context independence" property of the script can be checked during de-serialisation +or by a single traverse of the ErgoTree. This can be done both _off-chain_ (in the client +side dApp) and _on-chain_ in the Ergo node by simply checking that the tree doesn't contain +forbidden (context related) operations. + +Thus, if the dApp uses such "context independent" contracts, then it can predictably (and +deterministically) compute the `tx.accCost` value of verification cost of the +transaction. In this case neither the client nor the network bear the risks of +unexpectedly costly transaction execution and subsequent rejection by the network (wasting +time and resources). Note, the cost is computed as part of the transaction signing, which +should be done by the sender in any case. + +Thus an application can: + 1) Evaluate scripts and obtain `tx.accCost` value. + 2) Choose `costPrice` value and store it in `tx.feeOut.R4`. + 3) Compute `tx.feeOut.value = tx.accCost * costPrice`. + 4) Sign the transaction and send it to the network. + +Because all the scripts depend only on the transaction content, every verifier will always +finish `tx` verification with the same `tx.accCost` value. This means the transaction will +never be rejected as _over-limit_. This mechanism is transparent for the client dApp +implementation and no special ceremony has to be performed by the application. +Existing applications, however, have to be upgraded to use this benefits. Before upgrade +the transactions with missing `tx.feeOut.R4` value will be processed with [maximum +transaction cost limit](#maximum-transaction-cost). + +### New Costing of ErgoTree + +Ethereum's contracts are compiled into bytecode instructions which are then executed by +Ethereum VM as part of a transaction execution. Each instruction has an associated cost +(aka 'gas') and the cost of the whole transaction is accrued as instructions are executed +one by one until either transaction complete or `gasLimit` is reached. In the later case +the transaction is aborted and the `gasLimit * gasPrice` number of Wei is transferred to +the miner. + +In contrast, Ergo's approach is more light-weight and does not require using a _stateful_ VM +to execute contracts. Instead, the contracts are evaluated using _stateless_ interpreter and +thus many contracts can be evaluated in parallel. + +For each input box of the transaction being validated: +1) The contract (stored in a box) is deserialized into ErgoTree data structure ready +for direct and efficient execution by the interpreter. +2) A special Context data structure is created and passed to the interpreter +3) Interpreter evaluates the given contract in the given context. + +The operations above can be done in parallel for all inputs, this is because interpreter +never mutates (or changes) the blockchain state. +In addition, the interpreter doesn't have direct access to the blockchain, and rely +solely on the Context to evaluate contracts. + +The interpreter of the ErgoTree `t` is extended to accumulate the execution cost +`t.accCost` during evaluation all the tree nodes, and it checks that `t.accCost <= +costLimit` at any time. +Thus, upon completion of ErgoTree reduction, the result is a pair `(R, C1)`, +where `R` is a _Sigma Protocol Proposition Tree_ and `C1 = t.accCost` is the cost +accumulated during script reduction. +The resulting Sigma Protocol Proposition `R` must be further verified against proofs of +knowledge provided during transaction signing. The cost `C2` of `R` verification is +approximated by an Ahead Of Time costing without performing time-consuming crypto +operations. The resulting value `C1 + C2` is checked against `costLimit`. + +If at any time the accumulated cost exceeds `costLimit`, then an exception is thrown, +which is interpreted as an _over-limit script_ as the transaction is rejected. + +### References + +[1] The Extended UTXO Model. + Manuel Chakravarty,James Chapman,Kenneth MacKenzie,Orestis Melkonian,Michael Peyton Jones,Philip Wadler + January/2020, Workshop on Trusted Smart Contracts (Financial Cryptography 2020) + +[2] Demystifying Incentives in the Consensus Computer. + Luu, Loi & Teutsch, Jason & Kulkarni, Raghav & Saxena, Prateek. (2015). 706-719. + +## Appendix A: AOT vs JIT Costing + +### AOT Costing +Current implementation of block validation in Ergo (performed by every full node) implements +the first approach (i.e. AOT costing) shown in the following figure. +![AOT](eip5/AOT-costing.png) + +AOT cost estimation happens for every +[ErgoTree](https://ergoplatform.org/docs/ErgoTree.pdf) of every input box of every +transaction in a block. The total accumulated cost of all the input scripts in the +block is limited by the `MaxBlockCost` parameter which can be adjusted by miners. + +When a new block candidate is being assembled by a miner then for each new +transaction taken from the transaction pool the cost of running all input scripts is +being calculated. +The transaction is added to the block as long as the cost of all transactions in the block don't +exceed `MaxBlockCost` limit. When the cost is within the limit the transaction can be +added to the block candidate after it passes validation and all input ErgoTrees are +evaluated to `true`. + +AOT cost estimation predicts the cost of the script execution ahead +of time in a given context. This approach is often used in blockchains with UTXO transaction +model and a simple, non-turing-complete contract language. +The method work under assumption that cost estimation can be done much faster than +actual execution of the script. For non-turing-complete simple languages without looping +primitives, simple transaction context and only primitive types (like Bitcoin script) this +assumption is true. However for more complex languages like ErgoTree this is +increasingly not the case. The following is the list of limitations and drawbacks when the +AOT costing is applied to non-turing-complete (but still expressive language like ErgoScript). + +##### AOTC Pros +1) No need to decide on the appropriate value for `gasLimit` +2) Transaction sender doesn't loose money in case the transaction is rejected due to the exceeded cost. +3) Simple user-centric economy, where user only need to pay the fixed transaction fee, +defined dynamically by the current network usage (supply/demand ratio). + +##### AOTC Cons + +1) _It is complex_. Simple implementation is only possible for simple language (i.e. +primitive types, no collections, simple transaction context (again, no collections). +Adding collection types with at least the `map` operation leads to a much more complex +implementation if we still want to have expressive enough language. The `fold` operation +is even worse for AOT costing unless we strictly limit it to primitive aggregations. + +2) _It is in accurate_. We don't know the exact execution trace of the script +ahead of time, in particular we cannot decide which branch of `if` statement will be +executed (since we don't know the result of condition expression) when the condition +depends on blockchain context such as current block height. +As a result we have to estimate both branches and take the maximum. This inherent estimation +error is multiplied when this `if` is part of the `map` operation. A similar approximation +has to be done if the script accesses an element of a collection by an index. If the +elements have complex structure (like INPUTS or OUTPUTS collections of boxes in +ErgoScript), then we have to approximate the size of the element we want to access. Even if +we abstract this size as just a number of bytes (which is very naive, and it didn't really +worked in ErgoScript), this number may lie in the range from hundred to 4K bytes. Thus we +may have an order of magnitude over-estimation of the actual execution cost (or +underestimation if we decide to use optimistic approximation). + +3) _Low scalability_. Very limited number of transaction in a block due to +over-estimation. It turns out that poor cost approximation leads to over-estimation. +Because total cost of all transactions in a block must not exceed `MaxBlockCost` we can +put less transactions in the block which severely limits the maximum possible network +throughput. + +4) _High overhead_. To prevent rare spam attacks the miners need to do extra computations +for executing benign scripts. And complex implementation of AOT costing (for expressive +ErgoTree language and transaction context) makes this overhead comparable to or even +exceeding the execution of the benign scripts themselves. Yes, we can quickly reject +high-cost scripts, but have to pay the double price for that everyday. + +There are other technical difficulties, drawbacks and counter arguments for doing AOT +costing of contracts written in expressive language such as ErgoTree. However, the cons +list above should already be enough to motivate development of a better alternative, +combining advantage of both the extended UTXO model of Ergo and the simplicity of the JIT +costing. + +### JIT Costing + +An example of the JIT cost control (also called _dynamic_ cost control) is the Ethereum's +`gasLimit` checks performed during a transaction execution. + +In Ethereum, every transaction must specify an amount of _"gas"_ that it is willing to +consume (called _gasLimit), and the fee that it is willing to pay per unit gas +(_gasPrice_). At the start of the transaction execution `gasLimit * gasPrice` of Ether are +removed from the transaction sender's account. All operations during transaction execution +and every computational step taken by the virtual machine consumes a certain quantity of +gas. + +If a transaction execution processes fully, consuming less gas than its specified limit, +say with `gasRem` gas remaining, then the transaction executes normally, and at the end of +the execution the transaction sender receives a refund of `gasRem * gasPrice` and the miner +of the block receives a reward of `(gasLimit - gasRem) * gasPrice`. + +If a transaction "runs out of gas" mid-execution, then all execution reverts, but the +transaction is nevertheless valid, and the only effect of the transaction is to transfer +the entire sum `gasLimit * gasPrice` to the miner. + +JIT costing has the following pros and cons. + +##### JITC Pros +1) Additional Economic incentive for miners to execute contracts. The amount of purchase +from sender's account `(gasLimit - gasRem) * gasPrice` is payed to the miner, which +reflects the actual script execution costs. +2) Simple implementation using Cost Table for all VM commands and accumulation of the cost +along the execution trace. +3) Accurate approximation. Only actually executed operations are summed up. The execution +trace depends on the current state of the contract's data (i.e. current context) and +thus also the accumulated cost (i.e. `(gasLimit - gasRem)` value). + +##### JITC Cons +1) Transaction sender will loose funds in case of exceeding `gasLimit`. In a +turing-complete language of Ethereum VM (byte code instruction set) it may be hard to +choose appropriate `gasLimit` to associate with the transaction. This is in particular +due to the point 3) in the Pros list. As a result, the transaction will be considered +invalid if actual (and unpredictably big) gas consumption exceeds `gasLimit`, in which +case the sender looses the funds. +2) The actual complexity of contracts may be limited by `gasPrice`. +Since all computations require gas, the usability of the contracts becomes limited by not +the language (which is powerful) and not the actual computation expenses of the miners, +but by the current `gasPrice` asked by the miners. You would not want to spend more of your +funds for the gas required to execute a complex contract, than the amounts of funds you operate +with (i.e. sending $10 and pay $100 for the gas doesn't have economic sense). + +## Appendix B: Benefits of JIT costing + +1) When selecting transactions from the pool, a miner can quickly filter out transactions +that doesn't fit into `MaxBlockCost` using `costLimit = tx.feeOut.value / tx.feeOut.R4` when +R4 is specified. + +2) In many cases (such as P2PK contracts and others) when the cost of transaction +validation doesn't depend on the context the cost of `tx` may be exactly predicted by the +sender (see [Context-Independent Scripts](#context-independent-scripts)). +In such cases: + - when R4 is provided by the sender the transaction will not be rejected as over-limit + - existing applications (which don't set R4) can be supported (without change) by + either 1) using reasonable and small default values of `costLimit` for p2pk scripts + or 2) using pessimistic `MaxTxCost` as default of `costLimit`. + +3) In a long term, JIT costing enables an economic incentive to improve performance of +block validation including script evaluation. The miners are interested to put as much +transactions in every block as possible to maximize fee payments. +Considering the pool is full of transactions, the amount of transactions which can be put +into a new block is limited by a number of parameters: + - Block validation time is limited by 2 minutes tick interval between blocks. The + time budget for block assembly and validation itself is the small fraction of this time + (less than 5 seconds), as the block must also be distributed across the network and + validated by all nodes. This is necessary to avoid network splits. + - The maximum number of transactions in a block is limited by the `MaxBlockCost` + network parameter. This parameter can be adapted by miners via voting, as long as + the resulting blocks don't exceed the validation time limit. + - `MaxTxCost` initially is small and conservative, which limits the maximal possible + complexity of transactions. This allows to put more transaction in a block within a + fixed `MaxBlockCost`. Thus, in order to increase complexity of transactions + supported by the blockchain the following should be done: + 1) verifiers should improve computational efficiency of node software + 2) `MaxBlockCost` should be increased proportionally via voting + 3) `MaxTxCost` should be increased via voting + +Under these limits, miners and users as the community, are interested to improve throughput of block +assembly and validation software to process more transactions per second, with higher +total cost within the given time limits. The proposed JIT costing protocol for extended +UTXO model of Ergo enables a whole range of potential optimizations. This will be +especially important when the emission is over and the transaction fees along with storage +rent will be the only reward for miners. + + diff --git a/eip5/AOT-costing.png b/eip5/AOT-costing.png new file mode 100644 index 00000000..e6186961 Binary files /dev/null and b/eip5/AOT-costing.png differ diff --git a/eip5/jit-costed-tx-flowcard.png b/eip5/jit-costed-tx-flowcard.png new file mode 100644 index 00000000..cbe0aecf Binary files /dev/null and b/eip5/jit-costed-tx-flowcard.png differ