[WIP] BOLT14: Sharing x bit of Liquidity Information in the Friend of a Friend Network of Sender and Receiver

[renepickhardt]

As mentioned in December 2019 on the Mailinglist I propose to work on a new BOLT 14 which should be a collection of recommendations that help nodes to improve their abilities to discover payment paths with sufficient liquidity.

This proposal is a Work in progress and tries to spec out the foundation for nodes to be able to share balance information with their friends. It is important that the query can only ask for balance information of channels from its own channel partners. The learnt / shared information is not gossiped to other peers.

The design is currently in a way that a node can send a query_foaf_balances message which includes an amount and a flow direction to signal that it wishes to initiate a rebalance operation of the amount in a certain direction. The receiving node of this query has the option to respond with a list of short channel id's that it is connected to along which it would support a circular rebalancing operation with the given amount.

Having these two messages would lay the basis for

• a proactive rebalancing protocol
• a robust JIT routing implementation.

a few words to the privacy advocates who might oppose the idea of sharing information about funds:

1. Probing the balances of the friends of my friends is currently possible with alomost no effort and no cost. thus we could also allow nodes to share this information with far less network communication
2. It was shown recently that a JIT routing by default implementation would prevent arbitrary probing attacks almost completely
3. I have Work in progress and still unpublished research results in which I measured (with the help of simulated payments) that overall the network learns actually less balance information if JIT routing and balance sharing is active in comparison to the current probing based approch where nodes learn quite a lot of balance information with every failed routing attempt.

Regarding Numbering and Naming:

• BOLT 14 mainly extends BOLT 04 (onion routing) and BOLT 7 (routing-gossip) So it made sense to me to suggest 14 as a number.
• ODD types of lightning messages 437 and 439 are currently used in my proof of concept implementation and throughout the BOLT. Of course if we find consensus for those kind of BOLT we would have to assign the respective even types. 437 was chosen as it is the first prime number in the correct semantic range whose checksum is 14.
• Feature bits: I believe the next Featurebits that would be needed for this kind of message are 20/21
• I stuck with the query / reply naming similar to BOLT 7
• BOLT 14 itself might eventually be better named BOLT14-routing-pathfinding.md but as this is currently not par of it I gave it the name that reflecs what it currently does.

Background:

There is a reference implementation as a proof of concept as a clightning plugin. The implementation and this PR weere mainly created on the Lightning Hacksprint in May 2020. While my research at NTNU is the main driver for this work I want to give a shoutout to @qrest for help with the reference implementation and @cdecker for helpful discussions and reviews of earlier versions of this proposed BOLT.

[renepickhardt]

Just realizing that the results from the foaf_query could also be used in BOLT11 routing hints.
In this way a person who wants to get paid could make the query and indicate 2 hop routs with sufficient inbound capacity.
On the other side a person who wants to send an onion could run the query and know their two hop distance.

assuming that many paths are only 4 hops long we have good chances to find an intersection.

[renepickhardt]

When finding optimal pamemt flows (C.f.: https://arxiv.org/abs/2107.05322 ) the uncertainty of channel balances seems to be the main reason for many retries which produce overall load to the network. Given the fact that local balances are known there is no uncertainty and cost (for the uncertainty) to transport sats to the peers of a sender (of course other costs like fees, cltv risks, htlcs,... Might not be 0). Thus we effectively study a multi source min cost flow problem by starting to probe the channels of our neighbors. As probing and htlcs are generally expensive and slow to set up I think this proposal would reduce the overall traffic and improve reliability. (I have not simulated / quantified this yet)

Also from a recipient end they could query their neibors for liquidity and Include blinded route hints similar to what @rustyrussell included to the offers PR #798. Only difference the blinded routes up to two hops would certainly work after liquidity was announced.

This would transform the liquidity finding problem into a multisource multi sink min cost flow problem where sources are in the friend of a friend network of the sender and sinks are in the friend of a friend network of the recipient. And we'd have no uncertainty of the transportability of sats in those networks.

Again haven't studied the effects yet but I would assume that this should produce a significant

• gain in payment reliability
• reduction of network traffic
• reduction in locked liquidity (in htlcs) over time

[TheBlueMatt]

If anything we probably should be thinking of going the other direction - failing payments which are near the channel balance to make balance probing more difficult.

…

On Jul 28, 2021, at 04:31, Rene Pickhardt ***@***.***> wrote:  When finding optimal pamemt flows (C.f.: https://arxiv.org/abs/2107.05322 ) the uncertainty of channel balances seems to be the main reason for many retries which produce overall load to the network. Given the fact that local balances are known there is no uncertainty and cost (for the uncertainty) to transport sats to the peers of a sender (of course other costs like fees, cltv risks, htlcs,... Might not be 0). Thus we effectively study a multi source min cost flow problem by starting to probe the channels of our neighbors. As probing and htlcs are generally expensive and slow to set up I think this proposal would reduce the overall traffic and improve reliability. (I have not simulated / quantified this yet) Also from a recipient end they could query their neibors for liquidity and Include blinded route hints similar to what @rustyrussell included to the offers PR #798. Only difference the blinded routes up to two hops would certainly work after liquidity was announced. This would transform the liquidity finding problem into a multisource multi sink min cost flow problem where sources are in the friend of a friend network of the sender and sinks are in the friend of a friend network of the recipient. And we'd have no uncertainty of the transportability of sats in those networks. Again haven't studied the effects yet but I would assume that this should produce a significant gain in payment reliability reduction of network traffic reduction in locked liquidity (in htlcs) over time — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[renepickhardt]

Are you suggesting to make lightning less reliable than it already is? If yes I disagree (c.f: @rustyrussell 's 27 tries to deliver 10k sats (effectively revealing the full balance of 26 failed channels) https://twitter.com/rusty_twit/status/1420565586552057862?s=19 )

Probing is a regular part of the trial and error loop of the payment process. We can only deliver a payment if we find sufficient liquidity (which as it is unannounced needs to be probed for). Latest when a payment is successfull we will have learnt something about the liquidity of the involved channels. Sharing some information voluntarily might potentially on average reveal less information overall (needs to be tested though)

Also note that the proposal allows nodes to not reveal their liquidity upon request

[renepickhardt]

I recently created a simulation that shows the effects of the ideas of this proposal with respect to reliability of payment delivery when

1. Sharing some information about the liquidity in the friend of a friend network of sender and recipient
2. globaly announcing x bits of information of liquidity information about ones channel.

I have not ported the code back to the python simulation package but there is an old commit where I demonstrated the consequences of such strategies

When using optimization for reliability and fees we could deliver a 0.5 BTC payment with 5 failed onions in 2.609 sec of computational time for 1581 ppm

OTPTIMIZE RELIABILITY AND FEES
SUMMARY:
Rounds of mcf-computations: 2
Number of onions sent: 41
Number of failed onions: 5 failure rate: 12.20%
total runtime (including inefficient memory managment): 2.609 sec
Learnt entropy: 166.14 bits
Fees for successfull delivery: 79085.210 sat --> 1581 ppm

When using optimization only for reliablity we could deliver a 0.5 BTC payment with 3 failed onions in 2.223 sec of computational time for 2314 ppm

OPTIMIZE FOR soly for RELIABILITY
SUMMARY:
Rounds of mcf-computations: 2
Number of onions sent: 38
Number of failed onions: 3 failure rate: 7.89%
total runtime (including inefficient memory managment): 2.223 sec
Learnt entropy: 157.24 bits
Fees for successfull delivery: 115733.700 sat --> 2314 ppm

When using optimization mainly for fees we could deliver a 0.5 BTC payment with 57 failed onions in 6.112 sec of computational time for 1399 ppm

OTPTIMIZE MAINLY FOR FEES
SUMMARY:
Rounds of mcf-computations: 5
Number of onions sent: 192
Number of failed onions: 57 failure rate: 29.69%
total runtime (including inefficient memory managment): 6.112 sec
Learnt entropy: 268.16 bits
Fees for successfull delivery: 69992.010 sat --> 1399 ppm

When using FOAF liquidity sharing and optimizing mainly for fees we could deliver a 0.5 BTC payment with 1 failed onions in 1.796 sec of computational time for 1490 ppm. (Note this is cheaper and faster than using optimally reliable and cheap payment flows)

OTPTIMIZE MAINLY FOR FEES use foaf uncertainty reduction
channels with full knowlege: 215
channels with 2 Bits of less entropy: 8750
SUMMARY:
Rounds of mcf-computations: 2
Number of onions sent: 76
Number of failed onions: 1 failure rate: 1.32%
total runtime (including inefficient memory managment): 1.796 sec
Learnt entropy: 53.58 bits
Fees for successfull delivery: 74503.000 sat --> 1490 ppm

Also probably surprisingly using FOAF liquidity sharing is for reliablity more practical than globally reducing the uncertainty (either by probing or sharing some hints via gossip) where we optimized mainly for fees we could deliver a 0.5 BTC payment with 19 failed onions in 4.032 sec of computational time for 1417 ppm.

OTPTIMIZE MAINLY FOR FEES use global uncertainty reduction
SUMMARY:
Rounds of mcf-computations: 3
Number of onions sent: 236
Number of failed onions: 19 failure rate: 8.05%
total runtime (including inefficient memory managment): 4.032 sec
Learnt entropy: 169.60 bits
Fees for successfull delivery: 70887.930 sat --> 1417 ppm

TL;DR: Here is a table of all experiments

                   | failed onions |  ppm  | computational time [sec]  
-------------------------------------------------------------------------- 
reliable & cheap   |            5  |  1581 |        2.609  
-------------------------------------------------------------------------- 
only reliable      |            3  |  2314 |        2.223  
-------------------------------------------------------------------------- 
cheap              |           57  |  1399 |        6.112  
-------------------------------------------------------------------------- 
FOAF               |            1  |  1490 |        1.796   
-------------------------------------------------------------------------- 
global information |           19  |  1417 |        4.032    

From such results it seems the the friend of a friend sharing of liquidity information seems to give us the best of both worlds. In particular I think this may be relevant as the uniform prior that is assumed in optimally reliable and cheap payment flows may not hold as indicated by @bitromortac in this issue and as explained in the blog article shared in a recent post to lightning-dev. I think even small hints ins some regions about the liquidity will be very useful if liquidity is note uniformly distributed across the network.

Disclaimer: While this example is based on a very small sample / size statistic I can report that anytime I was running the code it seemed to have confirmed this picture so it shouldn't be cherry picking of a nice run. I just didn't have the time yet to conduct a proper evaluation

ps: As I think the term liquidity information is much more precise than balance I will rename the issue!

[tnull]

This is likely related, although they seem to only look at the global case. https://arxiv.org/pdf/1909.02717.pdf

[bshramin]

Is there something that prevents the node receiving the query from lying in response?

[otech47]

concept ack

the privacy vs reliability tradeoff is stark (as always). I can see privacy-first nodes closing channels with reliability-first peers if they find out they are advertising channel balance to others... but i think this proposal might have enough optionality to allay the privacy concerns?

Nodes are not required to respond to the query_foaf_balance message
Nodes MAY obfuscate the information in the reply_foaf_balance message

the spec might need a more details/examples of how a node might do this so it is easier to understand what the exact information leak risks are... it is worrying if this proposal increases the chances of a bifurcated LN where payments from non-FOAF nodes fail so much more than payments through FOAF nodes

could also be interesting to explore other versions of this same idea that can improve reliability (maybe less so) but with more obfuscated information-sharing... like instead of channel balance data, you can share average amounts/PPM routed across all channels in the past X days/weeks/months? is that any useful at all? or if you stick with channel-balance data but group them and say I have 100M sats across these 5 channels? or does that totally not help solve the problem?