From 5e9872dd0dd79a44c7b5769c7b3c5f94118c8615 Mon Sep 17 00:00:00 2001
From: Michael de Hoog <michael.dehoog@coinbase.com>
Date: Thu, 1 Feb 2024 19:36:56 -1000
Subject: [PATCH] Add FastLZ L1-Cost function specs

---
 specs/exec-engine.md       |  2 +-
 specs/fjord/exec-engine.md | 97 ++++++++++++++++++++++++++++++++++++++
 specs/fjord/overview.md    |  1 +
 3 files changed, 99 insertions(+), 1 deletion(-)
 create mode 100644 specs/fjord/exec-engine.md

diff --git a/specs/exec-engine.md b/specs/exec-engine.md
index 7aa4fc852..8c9a68fea 100644
--- a/specs/exec-engine.md
+++ b/specs/exec-engine.md
@@ -155,7 +155,7 @@ Where:
 - the computation is an unlimited precision integer computation, with the result in Wei and having
   `uint256` range.
 
-- zeoroes and ones are the count of zero and non-zero bytes respectively in the *full* encoded
+- zeroes and ones are the count of zero and non-zero bytes respectively in the *full* encoded
   signed transaction.
 
 - `l1BaseFee` is the L1 base fee of the latest L1 origin registered in the L2 chain.
diff --git a/specs/fjord/exec-engine.md b/specs/fjord/exec-engine.md
new file mode 100644
index 000000000..a41e56b0e
--- /dev/null
+++ b/specs/fjord/exec-engine.md
@@ -0,0 +1,97 @@
+# L2 Execution Engine
+
+## Fees
+
+### L1-Cost fees (L1 Fee Vault)
+
+#### Fjord L1-Cost fee changes (FastLZ estimator)
+
+Fjord updates the L1 cost calculation function to use a FastLZ-based compression estimator.
+The L1 cost is computed as:
+
+```
+l1FeeScaled = l1BaseFeeScalar*l1BaseFee*16 + l1BlobFeeScalar*l1BlobBaseFee
+l1CostSigned = (intercept + fastlzCoef*fastlzSize + uncompressedTxCoef*uncompressedTxSize) * l1FeeScaled / 1e12
+l1Cost = uint256(max(0, l1CostSigned))
+```
+
+Where:
+
+- the final `l1Cost` computation is an unlimited precision unsigned integer computation, with the result in Wei and
+  having `uint256` range.
+
+- the `l1CostSigned` calculation is an unlimited precision signed integer computation, with the result in Wei and
+  having `int256` range.
+
+- `l1BaseFee` is the L1 base fee of the latest L1 origin registered in the L2 chain.
+
+- `l1BlobBaseFee` is the blob gas price, computed as described in [EIP-4844][4844-gas] from the
+  header of the latest registered L1 origin block.
+
+- `l1BaseFeeScalar` is the L1 base fee scalar, scaled by `1e6` (see below for suggested values).
+
+- `l1BlobFeeScalar` is the L1 blob fee scalar, scaled by `1e6` (see below for suggested values).
+
+- `fastlzSize` is the length of the FastLZ-compressed RLP-encoded signed tx.
+
+- `uncompressedTxSize` is the length of uncompressed RLP-encoded signed tx.
+
+- `intercept`, `fastlzCoef`, and `uncompressedTxCoef` are constants calculated by linear regression (see next section),
+  scaled by `1e6`. These values can be negative.
+
+##### L1-Cost linear regression details
+
+The `intercept`, `fastlzCoef`, and `uncompressedTxCoef` constants are calculated by linear regression using a dataset
+of previous L2 transactions. The dataset is generated by iterating over all transactions in a given time range, and
+performing the following actions. For each transaction:
+1. Calculate the RLP-encoded transaction payload. Record the length of this payload as `uncompressedTxSize`.
+2. Compress the payload using FastLZ. Record the length of the compressed payload as `fastlzSize`.
+3. Compress the payload using a "best estimate" compression, defined as:
+   - Create two zlib streams, both using the maximum compression level.
+   - Bootstrap one of the streams with at least 64kb of transaction data (after compression).
+   - Write each transaction to both streams (flushing after each write), and record the size increase
+     of the larger buffer as `bestEstimateSize`.
+   - Each time the larger buffer grows larger than 128kb, empty the stream and swap the buffers.
+
+Once this dataset is generated, a linear regression can be calculated using the `bestEstimateSize` as
+the dependent variable and `fastlzSize` and `uncompressedTxSize` as the independent variables.
+
+The following Python code can be used to calculate the linear regression:
+
+```python
+import numpy as np
+from sklearn.linear_model import LinearRegression
+
+# Dataset format is:
+#   data               = [record...] (one or more records, concatenated)
+#   record             = bestEstimateSize ++ fastlzSize ++ uncompressedTxSize
+#   bestEstimateSize   = little-endian uint32
+#   fastlzSize         = little-endian uint32
+#   uncompressedTxSize = little-endian uint32
+
+dt = np.dtype([('best', '<u4'), ('fastlz', '<u4'), ('length', '<u4')])
+data = np.fromfile('./data.bin', dtype=dt)
+input_array = np.array(data.tolist())
+
+x = np.delete(input_array, [0], 1)
+y = input_array[:, 0]
+model = LinearRegression().fit(x, y)
+print(f'model: {model.intercept_} {model.coef_}')
+```
+
+As an example, we generated a dataset from all transactions on Optimism Mainnet in October 2023,
+and the resulting linear regression was:
+
+`-27.321890037208703 + 1.03146206*fastlzSize - 0.08866427*uncompressedTxSize`
+
+Scaling these values by `1e6` gives the constants used in the L1-Cost fee estimator:
+- `l1BaseFeeScalar = 11_111` (`~= 7600/0.684`)
+- `l1BlobFeeScalar = 1_250_000` (`~= 862000/0.684`)
+- `intercept = -27_321_890`
+- `fastlzCoef = 1_031_462`
+- `uncompressedTxCoef = -88_664`
+
+Note that the linear regression takes into account the current compression ratio, so the
+scalars `l1BaseFeeScalar` and `l1BlobFeeScalar` should be adjusted to account for any previous
+compression ratio they encoded. For example, if the previous compression ratio was `0.684`, then
+they should be divided by `0.684` to get the new scalars.
diff --git a/specs/fjord/overview.md b/specs/fjord/overview.md
index de1e3e730..1a9358f40 100644
--- a/specs/fjord/overview.md
+++ b/specs/fjord/overview.md
@@ -5,3 +5,4 @@ This document is not finalized and should be considered experimental.
 ## Execution Layer
 
 - [RIP-7212: Precompile for secp256r1 Curve Support](https://github.com/ethereum/RIPs/blob/master/RIPS/rip-7212.md)
+- [FastLZ compression L1 fee estimator](https://hackmd.io/@mdehoog/compression-analysis)