Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bolt 1: Specify that extensions to existing messages must use TLV #754

Merged
merged 4 commits into from
Mar 31, 2020
Merged

Bolt 1: Specify that extensions to existing messages must use TLV #754

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
6656b70
Bolt 1: Specify that extensions to existing messages must use TLV
t-bast Mar 2, 2020
70f1ed1
MAY close channels instead of MUST
t-bast Mar 12, 2020
67b342d
Explicit example of 0-length shutdown_script
t-bast Mar 12, 2020
a58cece
Extension not always possible for big packets
t-bast Mar 17, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 45 additions & 15 deletions 01-messaging.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ All data fields are unsigned big-endian unless otherwise specified.
* [The `ping` and `pong` Messages](#the-ping-and-pong-messages)
* [Appendix A: BigSize Test Vectors](#appendix-a-bigsize-test-vectors)
* [Appendix B: Type-Length-Value Test Vectors](#appendix-b-type-length-value-test-vectors)
* [Appendix C: Message Extension](#appendix-c-message-extension)
* [Acknowledgments](#acknowledgments)
* [References](#references)
* [Authors](#authors)
Expand All @@ -37,20 +38,12 @@ After decryption, all Lightning messages are of the form:
1. `type`: a 2-byte big-endian field indicating the type of message
2. `payload`: a variable-length payload that comprises the remainder of
the message and that conforms to a format matching the `type`
3. `extension`: an optional [TLV stream](#type-length-value-format)

The `type` field indicates how to interpret the `payload` field.
The format for each individual type is defined by a specification in this repository.
The type follows the _it's ok to be odd_ rule, so nodes MAY send _odd_-numbered types without ascertaining that the recipient understands it.

A sending node:
- MUST NOT send an evenly-typed message not listed here without prior negotiation.

A receiving node:
- upon receiving a message of _odd_, unknown type:
- MUST ignore the received message.
- upon receiving a message of _even_, unknown type:
- MUST fail the channels.

The messages are grouped logically into five groups, ordered by the most significant bit that is set:

- Setup & Control (types `0`-`31`): messages related to connection setup, control, supported features, and error reporting (described below)
Expand All @@ -61,10 +54,9 @@ The messages are grouped logically into five groups, ordered by the most signifi

The size of the message is required by the transport layer to fit into a 2-byte unsigned int; therefore, the maximum possible size is 65535 bytes.

A node:
- MUST ignore any additional data within a message beyond the length that it expects for that type.
- upon receiving a known message with insufficient length for the contents:
- MUST fail the channels.
A sending node:
- MUST NOT send an evenly-typed message not listed here without prior negotiation.
- MUST NOT send evenly-typed TLV records in the `extension` without prior negotiation.
- that negotiates an option in this specification:
- MUST include all the fields annotated with that option.
- When defining custom messages:
Expand All @@ -75,6 +67,21 @@ A node:
- SHOULD pick an even `type` identifiers when regular nodes should reject
the message and close the connection.

A receiving node:
- upon receiving a message of _odd_, unknown type:
- MUST ignore the received message.
- upon receiving a message of _even_, unknown type:
- MUST close the connection.
- MAY fail the channels.
- upon receiving a known message with insufficient length for the contents:
- MUST close the connection.
- MAY fail the channels.
- upon receiving a message with an `extension`:
- MAY ignore the `extension`.
- Otherwise, if the `extension` is invalid:
- MUST close the connection.
- MAY fail the channels.

### Rationale

By default `SHA2` and Bitcoin public keys are both encoded as
Expand All @@ -85,8 +92,10 @@ Length is limited to 65535 bytes by the cryptographic wrapping, and
messages in the protocol are never more than that length anyway.

The _it's ok to be odd_ rule allows for future optional extensions
without negotiation or special coding in clients. The "ignore
additional data" rule similarly allows for future expansion.
without negotiation or special coding in clients. The _extension_ field
similarly allows for future expansion by letting senders include additional
TLV data. Note that an _extension_ field can only be added when the message
`payload` doesn't already fill the 65535 bytes maximum length.

Implementations may prefer to have message data aligned on an 8-byte
boundary (the largest natural alignment requirement of any type here);
Expand Down Expand Up @@ -896,6 +905,27 @@ failure:
1. Invalid stream: 0xffffffffffffffffff 00 00 00
2. Reason: valid TLV records but invalid ordering

## Appendix C: Message Extension

This section contains examples of valid and invalid extensions on the `init`
message. The base `init` message (without extensions) for these examples is
`0x001000000000` (all features turned off).

The following `init` messages are valid:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That init message (along with the error) message as is, can't really be extended :/

It includes a variable sized byte slice that can occupy the entire message. If we really need to extend it in the future, we can create another init message.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you mean? Since that variable size byte slice starts with a u16 that explicitly sets its length, I believe it can be extended. For the specific case of init, it is even already extended with a TLV stream since we added the networks TLV. Can you clarify if I misunderstood?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah do you mean that the existing message may fill the 65535 bytes of the message? In that case definitely, it will hit the message size limit and fail. Do you think we should mention that the extension can only be done within the bounds of the message length limit?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I mean the feature vectors can fill the entire message. Adding a mention/note of this sounds good to me. In the typical case this shouldn't happen, but implementations should be ready for it.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, I'll add that.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in a58cece.

Did you really implement that many hidden features in lnd that you're afraid of filling up the init? 😆


- `0x001000000000`: no extension provided
- `0x00100000000001012a030104`: the extension contains two _odd_ TLV records (with types `0x01` and `0x03`)

The following `init` messages are invalid:

- `0x00100000000001`: the extension is present but truncated
- `0x00100000000002012a`: the extension contains unknown _even_ TLV records (assuming that TLV type `0x02` is unknown)
- `0x001000000000010101010102`: the extension TLV stream is invalid (duplicate TLV record type `0x01`)

Note that when messages are signed, the _extension_ is part of the signed bytes.
Roasbeef marked this conversation as resolved.
Show resolved Hide resolved
Nodes should store the _extension_ bytes even if they don't understand them to
be able to correctly verify signatures.

## Acknowledgments

[ TODO: (roasbeef); fin ]
Expand Down
48 changes: 28 additions & 20 deletions 02-peer-protocol.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,8 +123,13 @@ the funding transaction and both versions of the commitment transaction.
* [`point`:`htlc_basepoint`]
* [`point`:`first_per_commitment_point`]
* [`byte`:`channel_flags`]
* [`u16`:`shutdown_len`] (`option_upfront_shutdown_script`)
* [`shutdown_len*byte`:`shutdown_scriptpubkey`] (`option_upfront_shutdown_script`)
* [`open_channel_tlvs`:`tlvs`]

1. tlvs: `open_channel_tlvs`
2. types:
1. type: 0 (`upfront_shutdown_script`)
2. data:
* [`...*byte`:`shutdown_scriptpubkey`]

The `chain_hash` value denotes the exact blockchain that the opened channel will
reside within. This is usually the genesis hash of the respective blockchain.
Expand Down Expand Up @@ -207,9 +212,11 @@ The sending node:
- MUST set `channel_reserve_satoshis` greater than or equal to `dust_limit_satoshis`.
- MUST set undefined bits in `channel_flags` to 0.
- if both nodes advertised the `option_upfront_shutdown_script` feature:
- MUST include either a valid `shutdown_scriptpubkey` as required by `shutdown` `scriptpubkey`, or a zero-length `shutdown_scriptpubkey`.
- MUST include `upfront_shutdown_script` with either a valid `shutdown_scriptpubkey` as required by `shutdown` `scriptpubkey`, or a zero-length `shutdown_scriptpubkey` (ie. `0x0000`).
- otherwise:
- MAY include a`shutdown_scriptpubkey`.
- MAY include `upfront_shutdown_script`.
- if it includes `open_channel_tlvs`:
- MUST include `upfront_shutdown_script`.

The sending node SHOULD:
- set `to_self_delay` sufficient to ensure the sender can irreversibly spend a commitment transaction output, in case of misbehavior by the receiver.
Expand Down Expand Up @@ -291,8 +298,13 @@ funding transaction and both versions of the commitment transaction.
* [`point`:`delayed_payment_basepoint`]
* [`point`:`htlc_basepoint`]
* [`point`:`first_per_commitment_point`]
* [`u16`:`shutdown_len`] (`option_upfront_shutdown_script`)
* [`shutdown_len*byte`:`shutdown_scriptpubkey`] (`option_upfront_shutdown_script`)
* [`accept_channel_tlvs`:`tlvs`]

1. tlvs: `accept_channel_tlvs`
2. types:
1. type: 0 (`upfront_shutdown_script`)
2. data:
* [`...*byte`:`shutdown_scriptpubkey`]

#### Requirements

Expand Down Expand Up @@ -1127,16 +1139,15 @@ messages are), they are independent of requirements here.
* [`channel_id`:`channel_id`]
* [`u64`:`next_commitment_number`]
* [`u64`:`next_revocation_number`]
* [`32*byte`:`your_last_per_commitment_secret`] (option_data_loss_protect,option_static_remotekey)
* [`point`:`my_current_per_commitment_point`] (option_data_loss_protect,option_static_remotekey)
* [`32*byte`:`your_last_per_commitment_secret`]
* [`point`:`my_current_per_commitment_point`]

`next_commitment_number`: A commitment number is a 48-bit
incrementing counter for each commitment transaction; counters
are independent for each peer in the channel and start at 0.
They're only explicitly relayed to the other node in the case of
re-establishment, otherwise they are implicit.


### Requirements

A funding node:
Expand Down Expand Up @@ -1178,16 +1189,14 @@ The sending node:
next `revoke_and_ack` message it expects to receive.
- if `option_static_remotekey` applies to the commitment transaction:
- MUST set `my_current_per_commitment_point` to a valid point.
- otherwise, if it supports `option_data_loss_protect`:
- otherwise:
- MUST set `my_current_per_commitment_point` to its commitment point for
the last signed commitment it received from its channel peer (i.e. the commitment_point
the last signed commitment it received from its channel peer (i.e. the commitment_point
corresponding to the commitment transaction the sender would use to unilaterally close).
- if `option_static_remotekey` applies to the commitment transaction, or the sending node supports `option_data_loss_protect`:
- if `next_revocation_number` equals 0:
- MUST set `your_last_per_commitment_secret` to all zeroes
- otherwise:
- MUST set `your_last_per_commitment_secret` to the last `per_commitment_secret`
it received
- if `next_revocation_number` equals 0:
- MUST set `your_last_per_commitment_secret` to all zeroes
- otherwise:
- MUST set `your_last_per_commitment_secret` to the last `per_commitment_secret` it received

A node:
- if `next_commitment_number` is 1 in both the `channel_reestablish` it
Expand Down Expand Up @@ -1228,10 +1237,9 @@ A node:
- MUST NOT broadcast its commitment transaction.
- SHOULD fail the channel.
- otherwise:
- if `your_last_per_commitment_secret` does not match the expected values:
- if `your_last_per_commitment_secret` does not match the expected values:
- SHOULD fail the channel.
- otherwise, if it supports `option_data_loss_protect`, AND the `option_data_loss_protect`
fields are present:
- otherwise, if it supports `option_data_loss_protect`:
- if `next_revocation_number` is greater than expected above, AND
`your_last_per_commitment_secret` is correct for that
`next_revocation_number` minus 1:
Expand Down