Negative fees

[joostjager]

This change allows nodes to advertise a negative forwarding fee.

At the moment, there are no known pathfinding implementations that accept negative fees. This spec change is a first preparatory step towards that goal. A possible roadmap looks like this:

1. Switch fee data type to signed integer (this PR). When a node advertises a negative fee, existing implementations will interpret this as a very high positive fee. The channel will never be used because it is so expensive. Because of that, nodes won't advertise negative fees yet.

2. Implementations start parsing the fee as a signed integer, but convert negative numbers to zero to not break their pathfinding implementations. Nodes can advertise negative fees, but this is effectively the same as advertising zero fees. Those channels won't be skipped anymore as in phase 1.

   During forwarding, a negative fee is accepted. This means that the outgoing htlc amount may be less than the incoming htlc amount.

   Implementations that have a final route building post-processing pass such as lnd, custom pathfinders and users that build routes manually are able get some benefit from the negative fee already.

3. Implementations can choose to implement pathfinding algorithms that support negative edge weights such as bellman-ford to take maximum advantage of the negative fees.

The reason for making this spec change now is that it takes time to reach phase 2. Up to that point, changes are minor, but do unblock phase 3.

[t-bast]

Negative fees are a very controversial change that may impact the whole network and its incentives structure.
It deserves a thorough motivation and analysis, and the PR description doesn't contain that (or links to documents that fill this gap).
I'm not decided yet whether I'm in favor of this or not, but for now it's a concept NACK until this is properly motivated.

[joostjager]

I believe the only way to really learn about incentives is to experiment with it. Numerous routing node operators have told me that they need more control over flow, which for me is sufficient motivation.

I don't think interpreting fees as signed integers and rounding up to zero can do any harm. The alternative is to add another tlv field outbound_fee_discount, for nodes that care about it.

[TheBlueMatt]

I don't think interpreting fees as signed integers and rounding up to zero can do any harm.

It also doesn't allow anyone to experiment materially with negative fees. This only really makes sense if its going to be broadly supported.

[esneider]

Motivation: one of the strategies for rebalancing liquidity across channels is to advertise routing fees that will incentivize payers to make liquidity flow in the desired direction. If rebalancing some liquidity by other means costs X, it'd be rational for a routing node to pay up to X (ie. charge -X fees) for having someone make a payment that will rebalance the funds.

Gotchas:

• The slow speed of propagation of forwarding policy updates makes this rebalancing approach cumbersome.
• I'm unsure how we would prevent or deal with negative fee loops. Maybe a payment interceptor that would re-evaluate and potentially change a forwarding policy on the fly?

[joostjager]

I'm unsure how we would prevent or deal with negative fee loops.

I don't think this is a problem? If a nodes thinks it is worth x amount and charges for this -x to shift the balance by y, then it would also be willing to pay 2x for a 2y shift?

[NickRHill]

Negative fees are a very controversial change that may impact the whole network and its incentives structure. It deserves a thorough motivation and analysis, and the PR description doesn't contain that (or links to documents that fill this gap). I'm not decided yet whether I'm in favor of this or not, but for now it's a concept NACK until this is properly motivated.

I would be interested to know why the implementaiton of negative fees would be controversial.

From a node operator personal perspective, if such a change were implemented in a backwardly-compatble way, it would be their own decision as to whether they either charged a negative fee, or accepted a negative fee from another node operator. There is no compulsion for anyone to take part in negative fees.

From a network perspective, a negative fee system appears as though it would send pricing information to adjacent nodes which is currently not available. If I chose to provide a negative price to an adjacent node, it would invite them to connect to me, and be paid for looping their liquidity through my node. This would surely be appreciated by everyone who considers a functioning lightning network to be a good thing. The person looping their capital gets paid, my node gets balanced, and I then have capital to pass bitcoin on to the next hop.

It appears to me to be the solution to the flow problem (or in my hydrodynamic analogy on #921 the node price "pressure" problem). If negative fees aren't implemented, then the next solution to the flow problem would be the implementation of a channel opening regime which by default is double funded.

If we have a negative fee system, the need for double funded channels to maintain cross-network liquidity is probably eliminated.

[NickRHill]

If we agree that a negative fee system is a good thing, then implementation of this change at gossip level in a backwardly compatible way could be achieved by reserving bit 2 on channel_flags

https://github.com/lightning/bolts/blob/master/07-routing-gossip.md#the-channel_update-message

* [`byte`:`channel_flags`]

Bit Position	Name	Meaning
0	direction	Direction this update refers to.
1	disable	Disable the channel.
2	neg_ppm	fee_proportional_millionths should be multiplied by -1

[joostjager]

@NickRHill I don't think your proposal is backwards compatible.

[NickRHill]

@NickRHill I don't think your proposal is backwards compatible.

You are much more experienced than me on these matters so would love to understand why.

[joostjager]

When old clients do not look at the neg_ppm flag and interpret the fee as positive, they are unlikely to pick that node (depending on how negative the fee is). A backwards compatible scheme would probably set the fee to zero for older clients, and store the negative fee elsewhere where newer clients can find it.

[NickRHill]

When old clients do not look at the neg_ppm flag and interpret the fee as positive, they are unlikely to pick that node (depending on how negative the fee is). A backwards compatible scheme would probably set the fee to zero for older clients, and store the negative fee elsewhere where newer clients can find it.

I agree. So it's fair to say it is backwardly compatible in that it doesn't require re-engineering of the protocol, the protocol packets can still be exchanged, and positive or small negative PPM would rout with relatively little change.

Large negative fees would be counterproductive until the majority of network nodes implement the change.

We would recommend to node operators to avoid using large negative numbers until the majority of nodes are upgraded. Nodes accross the network will be incentivised to upgrade to take advantage of the lower routing fees.

[joostjager]

The alternative that doesn't have this restriction is just to add a tlv extension to the channel update that contains the negative fee, similar to how inbound fees are communicated.

[TheBlueMatt]

There's the further alternative of doing inbound fees at a local level - which is fully backwards compatible and old nodes even see the negative fee and use it when routing.

[joostjager]

How is that related to negative outbound routing fees? I linked the inbound fees blip just to show the channel update tlv mechanism that can potentially be used to keep negative fees backwards compatible.

[NickRHill]

I'm not familiar with such TLV extensions. You are much more experienced with the enginerring side of the protocol.

Are there downsides to a TLV extension? For example, is the payload made significantly larger, does it create any incompatibilities, and do all implementations have the ability to read TLV data without significant code and test work?

In essence, the suggestion appears to be to move the fee rate out of the channel update header altogether. If that is the idea, then perhaps the fee rate should also be expressed in parts per billion. A signed int32 can represent roughly +/- 2 billion and a PPB wouldn't ever sensibly approach a billion, which would be parity. PPM figures in single digits are sometimes seen today, so fractions of a PPM should be allowable in the protocol.

The signed int32 would be simply divided by 1000 to give PPM.

[NickRHill]

Which approach will attract more resistance?

[NickRHill]

El Salvador adopted Bitcoin as legal tender. My natural presumption was that LN was ready for that job. I had previously used it in late 2019 and had an awful experience. Lack of routeability often left me unable to make a payment, and when I could, with stupidly high fees, I'd have been better off sending on-chain.

While the network has much more liquidity now, very little of it can be used to benefit potential volume end users. IMO, LN is about 1% of where it should be today. In the absence of good arguments to the contrary, I think we can say we know exactly why the situation persists. When it comes to improving the network, we need to ignore rent-seekers and do what is best to make the network efficient for end-users.

With this in mind, if we can agree on an approach for negative fees, and the code is in trunk by the end of January,  functional at the protocol level, and accessible via lncli, I'll pay a bounty of 5Msat. 

Conditions: I'll start using a trunk-built LND node in late January 2023 and pay in the first week of February 2023. The node must receive and understand the negative PPM fee signal via normal signalling and in normal time and use it if available without special compilation flags or command line arguments. I should be able to see the negative fee charged by at least one hop by calling lncli buildroute with --amt and --hops as the only parameters, and that same route will be used to pay the bounty. 

The person who made the commit may post the invoice here.

I reserve the right to withdraw the bounty up to 7th January 2023 based on a compelling argument why the inclusion of a negative PPM will not improve the experience that end users will enjoy from using LN. Before making a counter-argument, I invite you to read my hydrodynamic analogy on #921 . 

[joostjager]

Are there downsides to a TLV extension? For example, is the payload made significantly larger, does it create any incompatibilities, and do all implementations have the ability to read TLV data without significant code and test work?

I don't think there are downside to this approach. The total size of the channel update increases a negligible amount, and routing for older clients remains as good as it can be.

Extra work is required either way.

[NickRHill]

One argument against negative fees is that a node, perhaps via another node, could find a bi-directional negative fee path, then oscillate payments repeatedly in either direction as quickly as the network will allow, thereby gaining the negative fee and draining the node.

To prevent this, a node would need to provide no more than one negative fee channel at a time, and the magnitude of that negative fee should not significantly exceed the magnitude of the positive fee of the next cheapest channel. This significantly limits the potential benefits of a negative fee system.

[NickRHill]

Perhaps a solution is to make an explorer which calculates and groups nodes based on mean price pressure. Then recommend nodes people may wish to connect to with costs for outbound liquidity to specific nodes based on median channel pricing. Such information will modify behaviour in such a way as to flatten the price pressure component of the graph. I reckon I can put something like that together in node.JS in a few days. Polishing it and making a nice web front end will take much longer.

Might be better to make an API and release source code so that an existing explorer can add the data to their metrics and put together a crude API browser?

[t-bast]

This has moved to bLIPs for now (lightning/blips#18 and lightning/blips#22), so I'm closing this until the bLIP experiments provide enough results to motivate (or not) a change to the BOLTs.