Self-adjusting onboarding fees with demand pricing

[anorth]

The proposal to separate proof fees from execution gas proposes removing the batch balancer mechanism, instead charging SPs an explicit fee for proving or updating a sector, separate from the gas cost of the on-chain proof verification. As technological advances reduce the hardware (off-chain) and validation (on-chain) costs, and expand the availability of blockspace, the Filecoin network should continue to capture revenue based on the core functionality of network-validated, cryptographically proven storage.

This proposal expands on that idea with a self-adjusting price mechanism, restoring a feature that the batch balancer offered before the introduction of FVM. This idea is a collaboration with @nicola.

The batch balancer was ok pre-FVM

The batch balancer mechanism had an element of demand pricing in its batch fee formula. The per-sector batch fee increases with base fee so that the total gas+fee cost of a proof increases faster than the base fee. Before the introduction of the FVM, the primary demand for blockspace is storage onboarding. This means the primary determinant of base fee is the onboarding rate and so this introduces a feedback mechanism where high onboarding rates would increase the fees associated with onboarding both via the gas cost (which could be countered by aggregation) and the batch balancer fee.

After the FVM is introduced, there will be many demands for blockspace which will change over time. The base fee will cease to be driven primarily by onboarding rate, and may fluctuate wildly. There is significant concern that volatile gas prices will price out network maintenance operations (Window PoSt) and increase costs and risks of onboarding. Technological advances that reduce the computational cost of proof verification would also further weaken the relationship between base fee and onboarding rate.

Returns depend on off-chain factors

SPs returns depend on income and costs denominated in both Filecoin tokens and fiat currencies. Mining rewards and on-chain fees are denominated in FIL, but factors like the cost of hardware, energy, and staff are usually fiat. An SP’s returns thus depend on the Filecoin token price and local costs of inputs, as well as on-chain ROI.

The prevalence of non-FIL costs makes it hard to determine the “right” price for a storage proof. The Filecoin network should capture some part of an SP’s expected return, but that return depends heavily on factors that are not observable on-chain. If the price of a proof is set too high relative to off-chain realities like macroeconomic conditions, energy costs, or the token price, storage providers will withdraw. But if the price is set too low after technological improvements or market euphoria, the Filecoin network will forgo significant revenue that it could have captured.

Demand pricing

The original proof fee proposal suggested setting the price by governance action, a slow and impedance-prone process. As off-chain factors vary, the price would almost certainly be “wrong” most of the time.

We could instead price onboarding dynamically at the right level to both ensure sustainable SP returns and capture value as those returns improve by setting the price based on demand, as evidenced by the recent historical storage onboarding rate.

Proposal: establish a per-sector onboarding fee in proportion to the recently (~days/weeks) observed net raw-byte power growth rate. A similar fee would apply at Replica Update, and we may want to monitor the rate of replica updates in order to inform the dynamic price. This fee should also vary with sector size and probably commitment duration.

A demand-based onboarding fee would automatically adjust to changes in both on-chain and off-chain factors, and expectations thereof, influencing SP returns. SPs would be expected to commit sectors up to the rate at which the fee reduces their returns to a sustainable level (these returns would still be quite large, reflecting an acceptable risk-adjusted return on such investment).

A stabilising influence

In times of low or negative network growth, a dynamic onboarding fee would reduce the Filecoin network’s value capture to a minimum (gas would still be charged), maximally encouraging additional onboarding. Zero network growth implies that SP expected returns are, in aggregate, at the minimum sustainable level so there is no room for the network to capture revenue without losing net commitments. Despite reducing revenue, encouraging additional commitments would have a beneficial impact on the token circulating supply, since the tokens locked up in pledge far exceed the amount burnt. In times of declining growth and increasing supply, the pledge from a marginal sector is more valuable than the foregone fee.

On the other hand, in times of strong growth, a dynamic onboarding fee would increase the network’s value capture automatically in response to increased SP demand for onboarding. The increased demand implies that SP returns are strong, and a fee based on onboarding rate would increase to capture part of those return. A demand-based onboarding fee would thus act as a stabilising influence on network growth rate, curbing the peaks (while capturing strong revenues) and boosting the troughs, as compared to a fixed fee.

All of this should be decoupled from execution gas scarcity, which will become increasingly uncorrelated with storage demand and thus destabilising.

Response to technological improvements

As a concrete and not-too-hypothetical scenario, technological improvements currently in development may reduce the costs of proving sectors significantly. These improvements will reduce both the on-chain proof verification cost, and a storage provider’s hardware and energy costs for proof generation. A reduction in FIL-denominated on-chain costs could be accounted for in a fixed fee, but the reduction on off-chain costs is very hard to price in FIL. The “right” reduction depends on the proportion of SPs’ total costs that the proof generation represents, as well as the Filecoin token price at which to translate those fiat savings into on-chain fees.

A demand-based onboarding fee would automatically balance these factors. When off-chain costs decrease, SP returns would improve and so SP’s may be expected to commit more sectors in order to exploit those returns. An increased onboarding rate would push the fee up until it had captured a sustainable share of the gains.

Target setting

This proposal introduces a different network parameter to be set by governance action. Instead of setting an onboarding fee explicitly, we must instead decide form and slope of the “fee market” function. The simplest reasonable function is probably linear with a floor (like the batch balancer), in which case parameters are:

• the minimum sector onboarding fee
• the rate of network raw byte power growth above which the fee increases
• the slope of the increase in fee with increased onboarding

Additional decisions include how to vary the fee with sector size (probably linearly) and commitment duration.

The reward baseline function could provide a good guide here, and it seems reasonable to couple the two concepts (so that these values change if the baseline function changes in the future). For example, we could set parameters so that the function passes through the current batch balancer floor (16M atto-FIL) when the growth rate matches the baseline function, but has a floor somewhat lower in order to boost onboarding below that point.

[AxCortesCubero]

I just want to point out the similarity between what is proposed here and the adjustable target block size mechanism from (#515).

Let’s first separate the two ingredients here, one which is about separating message kinds into different lanes, and the other is about introducing an explicit minimum base fee.

If we were doing only the second part, only adding minimum base fee (for all kinds of messages, without distinguishing), this is completely equivalent to adjusting the target block size. Moving the target block size (which is currently in principle set to be the largest that can be consistently processed) is only a specific mechanism that achieves increasing a minimum base fee. So talking about increasing minimum base fee, or reducing target block size results in the same thing. What I like about the adjustable target block size approach is that for me it is more transparent and interpretable.

Importantly the adjustable target block size proposal doesn’t touch the Maximum block size, so it is not limiting the actual maximum capacity of the block, only economically guiding the expected block size through an increase in base fee.

The way I had proposed the adjustable target block size, I deliberately did not specify what criterion was use to adjust the target block size. Part of my discussion was trying to get different Filecoin stakeholders to agree on what is it that Filecoin wants to maximize for, this is what I there call a target function. My idea is that the target block size can be adjusted with a transparent mechanism whose goal is to maximize whatever target function we decide.

When interpreting your proposed mechanism in this light, you are using an implicit target function to set your minimum base fee, which seems a bit counterintuitive to me. If you are saying minimum base fee should be increased when there is more onboarding, or equivalent target block size should be reduced when there is more onboarding, this means the target function you are using is something like (1/Onboarding rate). That is you are designing a mechanism that tries to minimize the onboarding rate (which doesn’t sound like a good thing).

If for instance you used the opposite target function (you tried to maximize onboarding rate), there would still be a stabilizing effect, in terms of the base fee remaining relatively stable with changes in onboarding demand. If there is a surge in demand for onboarding, this would lead to adjustable target block size growing, but this doesn’t mean base fee will actually be lower, as the lowering of the minimum base fee is offset by the actual increase in demand that brings the base fee back up.

The other element here is gas lanes. This is not something that was part of my original proposal, but definitely something we are considering for the next iteration (we were thinking more in terms of storage-related messages vs general FVM). The eventual form I had in mind would be two parameters, an overall target block size, and a parameter for the allocation of gas available in the block between storage and FVM messages.

Otherwise I think we are both generally circling around very similar ideas. The most productive path I see moving forward is for us to work together on the next iteration, taking the best parts from both our proposals.

[anorth]

Thanks @AxCortesCubero, I think yes and no. There is certainly some thematic alignment here, but my proposals are based on a core piece which your comments seem to talk past: that proof fees be separated from gas charges, as they are (now with FVM) aimed at very different things which will become de-correlated. This is not a "lane", because proofs are not a scarce resource like chain computation. This proposal is then about how to set the value of proof fees independent of gas.

Once we separate these, I become quite open to improvements in the gas market, including your proposal for dynamic block capacity. But that's a totally different topic to what I'm addressing here. We can talk about the relative benefits of adjusting the minimum base fee vs dynamic block sizing in your #515.

If you are saying minimum base fee should be increased when there is more onboarding, or equivalent target block size should be reduced when there is more onboarding, this means the target function you are using is something like (1/Onboarding rate). That is you are designing a mechanism that tries to minimize the onboarding rate (which doesn’t sound like a good thing).

This is what the current batch balancer mechanism does! However much as I dislike it, I think it's a mischaracterisation to say it's trying to minimise onboarding. Both it and my proposal are intended to improve value capture from onboarding, in proportion to the value that is available to be captured. Our signal for this value-availability is the demand for onboarding at a given price.

The other element here is gas lanes.

Just for clarity, I am not talking about gas lanes here. The only reason I mention gas at all is to highlight current problems and try to decouple something from gas. I do think gas lanes could be a good idea, but I am proposing something different that's not a gas model at all.

[AxCortesCubero]

It seems to me the main difference you talk about is more in the motivation, but ends up in mechanisms that work the same way.

I understand the motivation that proofs can have an explicit cost, It may be good for Filecoin that SP's still have to pay a cost for their proofs, even if in the future gas costs become very low. The problem is that right now the only way proofs are paid for is by their gas consumption, proofs that require more gas have to pay more. Let's say in the future their is too much block space, so that gas prices drop, but we still would want proofs to cost something, so your solution is explicit proof fees.

That makes sense but I think would be equivalent to having a gas lane for proofs, and limiting the block space (even though now theres new technology that allows for larger bock space) through an adjustable target block size for it. This would equivalently mean that SP's have to pay for their proofs, if gas fees become too low. So I think the difference is in motivation, and more philosophical, but results in the same mechanism.

because proofs are not a scarce resource like chain computation.

I'll say it is not super clear to me what you mean by this. I mean currently proofs are scarce because they consume gas. So unless you have an infinite sized block, proofs consume a scarce resource. If gas becomes no longer scarce, but you introduce an explicit proof fee, you turn proof back into a scarce resource (you are implicitly selecting an expected block size which is not the default maximum block size, when you introduce an additional fee).

Another curiosity, do you have any particular reason to single out onboarding proofs, vs recurrent sector proofs. Do you want to apply an explicit fee for onboarding a new sector, but not for later on proving that sector every day? Any particular reasoning for this?

This is what the current batch balancer mechanism does! However much as I dislike it, I think it's a mischaracterisation to say it's trying to minimise onboarding. Both it and my proposal are intended to improve value capture from onboarding, in proportion to the value that is available to be captured. Our signal for this value-availability is the demand for onboarding at a given price.

I think I disagree on what batch balancer is doing (I also don't like it, specially because of this reason, lack of transparency, hard to figure out what it is doing). I think batch balancer is more in line with trying to maximize network revenue (debatable if it is successful at this). Currently demand for proofs can only be measured by demand for gas usage (they are almost equivalent at the moment, since proving is most of the activity on filecoin). So batch balancer decides whether to open or close capacity, based on current base fee (demand for gas). If the base fee is too low, batching is artificially made more expensive than single proofs (capacity is reduced, fee per proof is artificially increased). If the base fee passes the batch balancer threshold, batching becomes the rational economic choice, capacity is opened up, fees stop being artificially increased (they are still organically increased because demand was high already). This produces a kind of stabilty of proof fees, they don't vary that much between low demand and high demand periods, as they would if there wasnt a batch balancer.

[DecentCr8]

It can be challenging to grasp the concept of adjustable target block size. Typically, the target block size is set based on the capacity of the network, taking into account factors such as the performance and capacity of the computing power, network transmission latency, data access rate, and distribution degree at the current network. The target block size may be adjusted as technology evolves, but it is not based on network usage.

Furthermore, setting a minimum base fee is not a priority. Our main concern is ensuring the sustainability of the system, even during periods of heavy congestion, rather than maximizing network revenue during idle times. Our goal should be to foster a thriving ecosystem that keeps the network busy, rather than trying to maintain a steady network revenue when the network is shrinking.

[anorth]

I agree we have different motivations and (if one introduces gas lanes with dynamic limits) can end up with mechanisms of similar impact. But the mechanisms are still not the same, and aligning with the mechanism directly with the motivation is valuable for understanding and as a base for future development.

I'll say it is not super clear to me what you mean by [proofs are not a scarce resource]. I mean currently proofs are scarce because they consume gas. So unless you have an infinite sized block, proofs consume a scarce resource. If gas becomes no longer scarce, but you introduce an explicit proof fee, you turn proof back into a scarce resource

Right now proofs consume some of the scarce resource that is computation, metered by gas. But if they hypothetically did not consume computation (e.g. aggregated infinitely for ~free), then they would not be scarce. An explicit fee does not create any maximum on proofs in a block. If the fee was demand-driven then eventually it might get high enough to practically set a limit, but this "limit" would be grounded in the profitability expectations of individual SPs (including their physical-world costs), not a protocol limit. The statement "you are implicitly selecting an expected block size which is not the default maximum block size, when you introduce an additional fee" is incorrect. I think this may be the key difference that leads you to currently think they are equivalent.

Computation gas is actually, physically limited by the computational power of validating nodes in the network. Even if gas prices go low, the computation throughput cannot sustainably increase, and cannot even transiently increase above 2x the defined limit. A gas mechanism is an appropriate tool for metering access to this physically scarce resource, but is not for a non-scarce resource like proofs.

If we used a gas lane to charge for proofs, then:

1. This would remain coupled to the computation cost of proof validation, and we'd have to adjust gas lane parameters every time validation technology improved
2. It would be very difficult to account simultaneously for two different proof technologies that had different validation costs (but the same value to SPs and the network). Gas consumption would not be proportional to value. This is a likely scenario when we introduce a new proof type without immediately disallowing the old.
3. There would be an actual limit on the number of sectors proven in any block, an artificial scarcity intended to adjust pricing
4. We would need to include a dynamic "block limit" mechanism to avoid creating an artificially constrained capacity long term
5. We would still need a batch balancer to set some per-proof price in the face of aggregated validation. A dynamic limit isn't enough.

I can see that a gas lane + dynamic limit + batch balancer can get close to the end result of demand-based pricing around a network growth target. But it's a very complicated and indirect mechanism to get there vs just setting an explicit fee schedule. The batch balancer part could be removed if it were a gas-lane-like mechanism that tracked proofs rather than computation units, but then we lose any benefits of re-using existing code.

In reality there is still some gas consumed to validate a proof. So in reality yes there's an upper limit on proofs per block determined by this, and proofs compete with other computation for the scarce resource. But we will strive to reduce this, and our protocol shouldn't rely on it.

I think your motivation behind dynamic gas limits on blocks is fine for its original target of execution gas prices, but is just stretched too far here. My key goal is to decouple proofs from execution gas, at which point we can then select the most appropriate mechanism for each without unwanted interactions (e.g. of the current batch balancer).

do you have any particular reason to single out onboarding proofs, vs recurrent sector proofs. Do you want to apply an explicit fee for onboarding a new sector, but not for later on proving that sector every day? Any particular reasoning for this?

No strong reasoning, I'm definitely open to alternatives here. I think the fee should be attached to the value provided to the SP (or that they transitively provide to clients). This value is storage x time. We could charge for the time component either up-front, by factoring it in to the onboarding proof fee, or ongoing as "rent" by adding a fee to Window Post, or just a periodic fee for maintained storage that's not coupled specifically to the action of proving. See this comment on the earlier discussion.

[steven004]

I'm supporting this as a general idea. Taking this into consideration, the discussion #430 I submitted can be closed.

There is some thoughts coming to my mind when I am thinking of the Per-sector on-boarding fee i proportion to the recently observed net raw-byte power growth rate mechanism.

Firstly, it should be no objections that the network resource consuming for proofs (for both new on-boarding, or updating a sector) verification should be still counted in the base-fee calculation, even though the base-fee are used for these messages;

And, we may encounter a situation that the cost of a sector on-boarding might be higher than the current one, which calculated with the mechanism via base-fee, no difference than other messages, which means this change may increase SP's cost under some scenarios. We may need to avoid this in design.

When it comes to the minimum sector on-boarding fee, same situation could happen since the base-fee could be really low (to 100attoFil), so, would the minimum sector on-boarding fee be a function, instead of a fixed number?

[anorth]

the network resource consuming for proofs (for both new on-boarding, or updating a sector) verification should be still counted in the base-fee calculation, even though the base-fee are used for these messages

Yes I agree with this. I think the calculation would be something like:

• compute the cost of the gas for proof verification, as the current base fee times a constant associated with the proof type and size
• subtract the gas cost from the target explicit onboarding fee
• burn that difference, if it is positive

In this way, the effective onboarding fee will not change as the base fee fluctuates, unless the base fee gets so high that the gas cost alone exceeds the target fee. The base fee would still affect the cost of state updates etc associated with onboarding. (And if we later implemented a separate gas lane for storage operations, these would be decoupled from gas demand from user contracts/actors. The gas lane's base fee would give short-term congestion control).

I think the minimum fee would probably be a fixed number though.

[juanpmcianci]

I’ll start by saying that I overall agree with (and like) this proposal.

Just to clarify/summarise/double check we are on the same page:

• Goal is to have dynamically-priced onboarding fees. These fees are meant to be a function of, e.g., {time-weighted average (TWA) of onboarding rate, duration, sector size, baseline}, but not base fee. At a first glance, I guess some directives for this function would include

  • TWA onboarding rate goes up -> fee goes up. This could in principle depend on the baseline.
  • smaller fees for higher durations
  • smaller fees for larger size (?) tbd.

• While onboarding fees should not depend on base fee, the opposite is not true; the gas used for onboarding should be counted in the EIP1559-like base fee computation.

If we agree with the above, then I guess the next step would be to construct several well-motivated candidates for the onboarding fee function, and then discuss.

It might also be worth investigating how —if at all— would this affect the message packing strategy of the miners.

• At the current state, miners are faced with a single constraint knapsack problem when choosing which messages to include.
• In the hypothetical case of gas lanes, for example, miners would be faced with a multiple-constraint knapsack problem, which is more computationally challenging than the single-constraint one.
• Given that the proposed mechanism is neither of these cases, It is not immediately obvious to me if introducing this proposed decoupling would have any effect in this problem formulation.

I’m also flagging that Chitra and Angeris (well-known in “academic” crypto) have recently ArXiVed a manuscript investigating something similar. I will take a deeper look at it to see if we can extract something meaningful for us.
https://arxiv.org/pdf/2208.07919.pdf

[anorth]

Thanks. We're mostly on the same page, except that I expect:

• larger fees for higher durations (possibly sublinear)
• larger fees for larger size (probably linear)

The fee scales with the value provided to SPs, or the value they can provide to clients. That value is the network's validation of the space x time of proven storage. For sector size, I think this is quite straightforward as a 2x larger sector earns 2x power (as well as supporting 2x deals). A practical constraint on doing this, though, might be to avoid changing the cost expectations of existing SP operations too much.

Similarly for sector duration, a 2x longer sector earns ~2x more reward (without an SDM) and can earn 2x more byte-epochs of deal fees from clients (including the FIL+ subsidy). However if we charge this fee up-front (vs ongoing as "rent"), it may be appropriate to discount a linear fee for time-value of money, or even further to value long commitments as a network good.

I think this largely independent of any duration multiplier proposal. But if such a multiplier is implemented, an at-least-linear fee per duration seems justified.

[adam-dltx]

I am supportive of this concept. As you've pointed out in #557, it is critical to separate the rivalrous and non-rivalrous aspects of the network. At a high level, the mechanism appears to be an effective way for the network to capture value no matter the market condition.

My only hesitation is that during phases of high storage growth onboarding fees may make deal-making unattractive. This scenario could lead to several outcomes:

• SPs may try to time onboarding according to the market which could lead to a natural disincentive to grow their businesses on their schedule. Timing the market could adversely affect clients and send them shopping elsewhere.
• A demand-based fee could strongly bias the largest, most cost-competitive providers, as they will be willing to make deals when the smaller shops cannot. Fees could compound with increased borrowing FIL interest rates to make smaller shops even less competitive. Eventually, this squeeze could reduce the decentralization of storage.

I will leave room for the fact that these may be desirable outcomes from certain perspectives. These fees may also be so small that my statements above are moot.

I am unsure if this is viable, but I will float the idea: It may be desirable to implement a fee system that treats deals of high urgency differently than non-urgent deals, creating an onboarding queue of sorts. Those SPs willing to pay higher fees get to the front of the line, whereas those willing to wait and pay less fees can onboard profitably.