Investigate unrealistic base cost of DataReceiptCreationConfig

[Longarithm]

In recent runtime-params-estimator launches, DataReceiptCreationConfig base cost is 50 TGas, which is unrealistically high comparing to current cost of 4.5 TGas.

Examples:
#4231
#4455

data_receipt_creation_config: DataReceiptCreationConfig {
  base_cost: Fee {
      send_sir: 50745691155500,
      send_not_sir: 50745691155500,
      execution: 50745691155500,
  }
  ...
}

cc @bowenwang1996 @olonho

[bowenwang1996]

To be clear, the current cost of 4.5Tgas is also unrealistically high. See #3279 for a discussion on this.

[Longarithm]

From the first sight, difference is explained by taking IO costs into account:
#3771 (comment) - 50 TGas
#3771 (comment) - 300 Ggas
My local computation in docker confirm this.

But there is conflicting info here:
#3771 (comment) - 40 Ggas in both cases.
Need to investigate further

cc #4474 @matklad

[Longarithm]

Data receipt fee estimation depends on its place in the vector of metrics: https://github.com/near/nearcore/blob/master/runtime/runtime-params-estimator/src/cases.rs#L536

Consider an example:

    let v = calls_helper! {
        data_receipt_base_10b_1000_1 => data_receipt_base_10b_1000,
        data_receipt_10b_1000_1 => data_receipt_10b_1000,
        data_receipt_100kib_1000_1 => data_receipt_100kib_1000
        cpu_ram_soak_test => cpu_ram_soak_test,
        base_1M => base_1M,
        read_memory_10b_10k => read_memory_10b_10k,
        ...
        data_receipt_base_10b_1000_2 => data_receipt_base_10b_1000,
        data_receipt_10b_1000_2 => data_receipt_10b_1000,
        data_receipt_100kib_1000_2 => data_receipt_100kib_1000
    };

Computation of DataReceipt fees based on data_receipt_base_10b_1000_1, data_receipt_10b_1000_1, data_receipt_100kib_1000_1 yields 30 Ggas; same computation for data_receipt_*_2 fees yields 50 TGas.

Also, if we cleanup data by re-creating RuntimeTestbed before computation of data_receipt_*_2 fees, we get 30 Ggas again.

Supposed explanation:

• initially storage is in optimal state A
• we compute data_receipt_*_1 fees and get 30 Ggas
• computation of some metric M moves storage to suboptimal state B
• we compute data_receipt_*_2 fees and get 50 Tgas

I currently suspect M = storage_write_10kib_key_10b_value_1k.

It makes sense to use separate RuntimeTestbed for each computation. But anyway we have to separate the following two cases:

• on mainnet, storage never comes to state B, thus 30 Ggas is the correct realistic price
• on mainnet, there is a way to put storage to state B, and then we risk to have x1000 underestimated DataReceipt prices for unknown period of time.

To do so, I plan to understand more deeply what exactly causes the difference by profiling tools.

cc @matklad @olonho @nearmax

[bowenwang1996]

@Longarithm is it correct that data_receipt_creation_cost measures the cost of one storage write? If not, what exactly does it measure?

[Longarithm]

There is also a curious dependency between fees and accounts number.
In the notation of #4482 (comment):

+----------+----------------------+----------------------+
| Accounts | data_receipt_*_1 fee | data_receipt_*_2 fee |
+----------+----------------------+----------------------+
| 10K      | 37 Ggas              | 34 Ggas              |
| 20K      | 35 Ggas              | 50 Tgas              |
| 50K      | 40 Ggas              | 120 Tgas             |
| 100K     | 37 Ggas              | 258 Tgas             |
+----------+----------------------+----------------------+

UPD: added row for 100K accounts

[Longarithm]

@bowenwang1996
Base cost measures cost of DataReceipt creation and processing: https://github.com/near/nearcore/blob/master/core/primitives-core/src/runtime/fees.rs#L71-L85
Strictly speaking, it measures cost of:

• set, get and remove for ReceiptData, e.g.

  nearcore/core/store/src/lib.rs

  Line 341 in 2c4f6dc

  pub fn set_received_data(

• set, get and remove for PostponedReceiptId, e.g.

  nearcore/runtime/runtime/src/lib.rs

  Line 925 in 28d9b7d

  set(

• update of pending data count

  nearcore/runtime/runtime/src/lib.rs

  Line 898 in 28d9b7d

  set(

Though I'm not entirely sure that all these operations are called ~1000 times, as we expect in the measurement.
It's possible that 1000 DataReceipts are processed before processing ActionReceipt which collects them, and in such case we don't save postponed receipt ids:

nearcore/runtime/runtime/src/lib.rs

Line 914 in 28d9b7d

ReceiptEnum::Action(ref action_receipt) => {

I need to double-check this. cc @olonho @matklad

Note that we don't consider bytes cost here, in which I see no discrepancies.

[bowenwang1996]

@Longarithm is the base cost calculated based on some trivial data or through some statistical method? It looks like this involves at most 4 storage operations and it should not be more expensive than 4 storage writes of [some trivial data]. Also it seems to me that this fee depends on the shape of the trie, but we also have a separate touching_trie_node fee that accounts for this.

[MaksymZavershynskyi]

This explains lots of discrepancies that we are observing: https://near.zulipchat.com/#narrow/stream/295306-dev-contract-runtime/topic/fees.20.26.20state.20size/near/247505050

[Longarithm]

Closing, because we have a decent explanation for the issue, and it was fixed in #4647.