Avoid storing a full FinalOnionHopData in OnionPayload::Invoice #1447

andozw · 2022-04-23T17:21:55Z

Splits these commits from #1221 and #1445 because they should be separate.

codecov-commenter · 2022-04-23T17:46:25Z

Codecov Report

Merging #1447 (fc77c57) into main (9bdce47) will decrease coverage by 0.00%.
The diff coverage is 80.55%.

@@            Coverage Diff             @@
##             main    #1447      +/-   ##
==========================================
- Coverage   90.90%   90.89%   -0.01%     
==========================================
  Files          75       75              
  Lines       41616    41624       +8     
  Branches    41616    41624       +8     
==========================================
+ Hits        37829    37834       +5     
- Misses       3787     3790       +3

Impacted Files	Coverage Δ
lightning/src/ln/channelmanager.rs	`84.70% <80.00%> (-0.05%)`	⬇️
lightning/src/ln/inbound_payment.rs	`93.49% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9bdce47...fc77c57. Read the comment docs.

vincenzopalazzo

LGTM!

TheBlueMatt · 2022-04-25T15:10:37Z

lightning/src/ln/channelmanager.rs

 											log_trace!(self.logger, "Failing HTLCs with payment_hash {} as the total value {} ran over expected value {} (or HTLCs were inconsistent)",
-												log_bytes!(payment_hash.0), total_value, $payment_data_total_msat);
-											fail_htlc!(claimable_htlc);


Gah, this is wrong and our test coverage is really lacking here. I'm working on some improved testing here but in the mean time the fail_htlc call has to stay and the htlcs.push calls need to go back in the branches instead of being unconditional.

Do you intend to resolve it in this PR before merging it?

@arik-so yes

valentinewallace

Looks pretty good! Do we wanna rebase it on #1451 to get the test coverage there?

valentinewallace · 2022-04-26T19:22:02Z

lightning/src/ln/channelmanager.rs

 	value: u64,
 	onion_payload: OnionPayload,
 	timer_ticks: u8,
+	total_msat: u64,


Could be nice to document the difference between value and total_msat

Agreed; in fact, would be necessary imo

Is this close to accurate?

/// HTLCs that are to us and can be failed/claimed by the user struct ClaimableHTLC { prev_hop: HTLCPreviousHopData, cltv_expiry: u32, /// The amount (in msats) of this MPP part value: u64, onion_payload: OnionPayload, timer_ticks: u8, /// The sum total of all MPP parts total_msat: u64, }

It was not super clear to me, but I am going to walk through the code more and try to figure it out.

That looks good to me!

andozw · 2022-04-27T04:06:16Z

Looks pretty good! Do we wanna rebase it on #1451 to get the test coverage there?

I'll give it a go

andozw · 2022-04-29T17:10:41Z

lightning/src/ln/channelmanager.rs

 														total_value = msgs::MAX_VALUE_MSAT;
 													}
 													if total_value >= msgs::MAX_VALUE_MSAT { break; }
 												},
 												_ => unreachable!(),
 											}
 										}
-										if total_value >= msgs::MAX_VALUE_MSAT || total_value > $payment_data_total_msat {
+										if total_value >= msgs::MAX_VALUE_MSAT || total_value > claimable_htlc.total_msat {


@TheBlueMatt - should this be || total_value > $payment_data.total_msat ? The tests pass either way. I haven't rebased against #1451 yet but it looks like that one might be ready to merge?

Yea, its definitely more readable that way. That said I believe they're the same, though its a good bit of careful reading to see that, and I'm not even 100% confident of that after reading it five times.

andozw · 2022-04-29T17:19:54Z

rebased

valentinewallace

One comment but otherwise ACK with Matt's suggestion. Appreciate that these commits were separated out!

valentinewallace · 2022-04-29T21:52:08Z

lightning/src/ln/channelmanager.rs

@@ -3465,7 +3476,7 @@ impl<Signer: Sign, M: Deref, T: Deref, K: Deref, F: Deref, L: Deref> ChannelMana
 						debug_assert!(false);
 						return false;
 					}
-					if let OnionPayload::Invoice(ref final_hop_data) = htlcs[0].onion_payload {
+					if let OnionPayload::Invoice { _legacy_hop_data: ref final_hop_data } = htlcs[0].onion_payload {


Not a blocker, but it looks like we can get rid of this usage as well by accessing htlcs[0].total_msat instead of final_hop_data? @TheBlueMatt

TheBlueMatt

Okay, sorry for the delay on this one, LGTM modulo two trivial comments.

TheBlueMatt · 2022-05-01T17:42:10Z

lightning/src/ln/channelmanager.rs

-	Invoice(msgs::FinalOnionHopData),
+	Invoice {
+		/// This is only here for backwards-compatibility in serialization, in the future it can be
+		/// removed, breaking clients running 0.0.104 and earlier.


heh, well that's stale - should read "0.0.106", assuming this lands for 107, which it should.

TheBlueMatt · 2022-05-01T17:51:02Z

lightning/src/ln/channelmanager.rs

-										(incoming_cltv_expiry, OnionPayload::Invoice(payment_data), phantom_shared_secret),
+								let (cltv_expiry, onion_payload, payment_data, phantom_shared_secret) = match routing {
+									PendingHTLCRouting::Receive { payment_data, incoming_cltv_expiry, phantom_shared_secret } => {
+										let _legacy_hop_data = msgs::FinalOnionHopData {


nit: replace the explicit instantiation with just let _legacy_hop_data = payment_data.clone(), given that's what we're doing anyway.

...instead of accessing it via the `OnionPayload::Invoice` form. This may be useful if we add MPP keysend support, but is directly useful to allow us to drop `FinalOnionHopData` from `OnionPayload`.

andozw · 2022-05-02T16:37:45Z

rebased

andozw · 2022-05-02T16:51:36Z

Addressed the 2 minor fixups as an amend on the last commit since they belong there and were trivial. Should be ready for another 👀

valentinewallace · 2022-05-02T17:53:05Z

lightning/src/ln/channelmanager.rs

 	/// Contains a total_msat (which may differ from value if this is a Multi-Path Payment) and a
 	/// payment_secret which prevents path-probing attacks and can associate different HTLCs which
 	/// are part of the same payment.


Sorry should've seen this sooner, not a blocker but I think this can be removed and just say "Indicates this incoming onion payload is for the purpose of paying an invoice" or something like that

So replace Contains a total_msat (which may differ from value if this is a Multi-Path Payment) and a payment_secret which prevents path-probing attacks and can associate different HTLCs which are part of the same payment.
with:
Indicates this incoming onion payload is for the purpose of paying an invoice ?

Yep, that sounds great.

We only use it to check the amount when processing MPP parts, but store the full object (including new payment metadata) in it. Because we now store the amount in the parent structure, there is no need for it at all in the `OnionPayload`. Sadly, for serialization compatibility, we need it to continue to exist, at least temporarily, but we can avoid populating the new fields in that case.

TheBlueMatt added the Seeking Code Review label Apr 24, 2022

vincenzopalazzo previously approved these changes Apr 25, 2022

View reviewed changes

TheBlueMatt reviewed Apr 25, 2022

View reviewed changes

TheBlueMatt mentioned this pull request Apr 26, 2022

Add test coverage for failure of inconsistent MPP parts #1451

Merged

valentinewallace reviewed Apr 26, 2022

View reviewed changes

andozw dismissed vincenzopalazzo’s stale review via d0cf6b8 April 29, 2022 16:33

andozw force-pushed the seana.20220422.avoid-storing-FinalOnionHopData branch from b4ba546 to d0cf6b8 Compare April 29, 2022 16:33

andozw commented Apr 29, 2022

View reviewed changes

andozw force-pushed the seana.20220422.avoid-storing-FinalOnionHopData branch from d0cf6b8 to 4da7916 Compare April 29, 2022 17:18

valentinewallace reviewed Apr 29, 2022

View reviewed changes

andozw force-pushed the seana.20220422.avoid-storing-FinalOnionHopData branch from 4da7916 to 16a6ad2 Compare April 30, 2022 16:32

TheBlueMatt reviewed May 1, 2022

View reviewed changes

TheBlueMatt removed the Seeking Code Review label May 1, 2022

TheBlueMatt added 2 commits May 2, 2022 09:37

Pass FinalOnionHopData to payment verify by reference, not clone

26c0150

Store total payment amount in ClaimableHTLC explicitly

62f8df5

...instead of accessing it via the `OnionPayload::Invoice` form. This may be useful if we add MPP keysend support, but is directly useful to allow us to drop `FinalOnionHopData` from `OnionPayload`.

andozw force-pushed the seana.20220422.avoid-storing-FinalOnionHopData branch from 16a6ad2 to 1973f1f Compare May 2, 2022 16:37

andozw force-pushed the seana.20220422.avoid-storing-FinalOnionHopData branch from 1973f1f to c611aff Compare May 2, 2022 16:49

valentinewallace previously approved these changes May 2, 2022

View reviewed changes

andozw dismissed valentinewallace’s stale review via fc77c57 May 2, 2022 18:41

andozw force-pushed the seana.20220422.avoid-storing-FinalOnionHopData branch from c611aff to fc77c57 Compare May 2, 2022 18:41

TheBlueMatt approved these changes May 2, 2022

View reviewed changes

TheBlueMatt assigned valentinewallace May 2, 2022

valentinewallace approved these changes May 2, 2022

View reviewed changes

TheBlueMatt merged commit e3a305c into lightningdevkit:main May 2, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Avoid storing a full FinalOnionHopData in OnionPayload::Invoice #1447

Avoid storing a full FinalOnionHopData in OnionPayload::Invoice #1447

andozw commented Apr 23, 2022

codecov-commenter commented Apr 23, 2022 •

edited

Loading

vincenzopalazzo left a comment

TheBlueMatt Apr 25, 2022

arik-so Apr 27, 2022

andozw Apr 28, 2022

valentinewallace left a comment

valentinewallace Apr 26, 2022

arik-so Apr 27, 2022

andozw Apr 28, 2022

valentinewallace Apr 28, 2022

andozw commented Apr 27, 2022

andozw Apr 29, 2022

TheBlueMatt Apr 29, 2022

andozw commented Apr 29, 2022

valentinewallace left a comment •

edited

Loading

valentinewallace Apr 29, 2022

andozw Apr 30, 2022

TheBlueMatt left a comment

TheBlueMatt May 1, 2022

TheBlueMatt May 1, 2022

andozw commented May 2, 2022

andozw commented May 2, 2022

valentinewallace May 2, 2022

andozw May 2, 2022

TheBlueMatt May 2, 2022

Avoid storing a full FinalOnionHopData in OnionPayload::Invoice #1447

Avoid storing a full FinalOnionHopData in OnionPayload::Invoice #1447

Conversation

andozw commented Apr 23, 2022

codecov-commenter commented Apr 23, 2022 • edited Loading

Codecov Report

vincenzopalazzo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

valentinewallace left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andozw commented Apr 27, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andozw commented Apr 29, 2022

valentinewallace left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TheBlueMatt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andozw commented May 2, 2022

andozw commented May 2, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-commenter commented Apr 23, 2022 •

edited

Loading

valentinewallace left a comment •

edited

Loading