Skip to content

Commit

Permalink
Clarifies that processProposal may be called for set of transactions …
Browse files Browse the repository at this point in the history
…different from the one returned in the preceding prepareProposal (cometbft#1033)

If a proposer fails after calling prepareProposal and before calling processProposal, then the following may happen upon restarting:
- if failed before signing another message, then will invoke prepareProposal again, sign a new block, probably empty, and propose it;
- if failed after signing a proposal but before writing the  proposal message into the WAL, then will invoke prepareProposal and produce a new, probably empty block, [fail to sign it](https://github.com/cometbft/cometbft/blob/2789a59a9cc61c6ea56a6b266eeadf0f26ca2456/consensus/state.go#L1221), and not invoke processProposal; prevote timeouts will ensure the CometBFT is not stuck;
- if failed after writing the proposal message to the WAL, then will invoke prepareProposal, produce a new, probably empty block, fail to sign it, and invoke processProposal with the block signed before crashing.

---

#### PR checklist

- [ ] Tests written/updated
- [ ] Changelog entry added in `.changelog` (we use [unclog](https://github.com/informalsystems/unclog) to manage our changelog)
- [x] Updated relevant documentation (`docs/` or `spec/`) and code comments
  • Loading branch information
lasarojc authored Jun 29, 2023
1 parent d31be6b commit b23ef56
Show file tree
Hide file tree
Showing 4 changed files with 94 additions and 76 deletions.
61 changes: 35 additions & 26 deletions spec/abci/abci++_basic_concepts.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ title: Overview and basic concepts

# Overview and basic concepts

## ABCI++ vs. ABCI
## ABCI 2.0 vs. ABCI

[↑ Back to Outline](#outline)

Expand All @@ -40,18 +40,18 @@ as the Application cannot require validators to do more than executing the trans
finalized blocks. This includes features such as threshold cryptography, and guaranteed IBC
connection attempts.

ABCI++ addresses these limitations by allowing the application to intervene at three key places of
ABCI 2.0 addresses these limitations by allowing the application to intervene at three key places of
consensus execution: (a) at the moment a new proposal is to be created, (b) at the moment a
proposal is to be validated, and (c) at the moment a (precommit) vote is sent/received.
proposal is to be validated, and (c) at the moment a (precommit) vote is sent/received.
The new interface allows block proposers to perform application-dependent
work in a block through the `PrepareProposal` method (a); and validators to perform application-dependent work
and checks in a proposed block through the `ProcessProposal` method (b); and applications to require their validators
to do more than just validate blocks through the `ExtendVote` and `VerifyVoteExtensions` methods (c).
and checks in a proposed block through the `ProcessProposal` method (b); and applications to require their validators
to do more than just validate blocks through the `ExtendVote` and `VerifyVoteExtensions` methods (c).

Furthermore, ABCI 2.0 coalesces {`BeginBlock`, [`DeliverTx`], `EndBlock`} into `FinalizeBlock`, as a
simplified, efficient way to deliver a decided block to the Application.

## Method overview
## Methods overview


[↑ Back to Outline](#outline)
Expand All @@ -60,7 +60,7 @@ Methods can be classified into four categories: *consensus*, *mempool*, *info*,

### Consensus/block execution methods

The first time a new blockchain is started, CometBFT calls `InitChain`. From then on, method
The first time a new blockchain is started, CometBFT calls `InitChain`. From then on, method
`FinalizeBlock` is executed upon the decision of each block, resulting in an updated Application
state. During the execution of an instance of consensus, which decides the block for a given
height, and before method `FinalizeBlock` is called, methods `PrepareProposal`, `ProcessProposal`,
Expand All @@ -82,29 +82,32 @@ call sequences of these methods.
can make changes to the raw proposal, such as modifying the set of transactions or the order
in which they appear, and returns the
(potentially) modified proposal, called *prepared proposal* in the `ResponsePrepareProposal`
call. The logic modifying the raw proposal can be non-deterministic.
call.
The logic modifying the raw proposal MAY be non-deterministic.

- [**ProcessProposal:**](./abci++_methods.md#processproposal) It allows a validator to
perform application-dependent work in a proposed block. This enables features such as immediate
block execution, and allows the Application to reject invalid blocks.

CometBFT calls it when it receives a proposal and _validValue_ is `nil`.
The Application cannot modify the proposal at this point but can reject it if it is
The Application cannot modify the proposal at this point but can reject it if
invalid. If that is the case, the consensus algorithm will prevote `nil` on the proposal, which has
strong liveness implications for CometBFT. As a general rule, the Application
SHOULD accept a prepared proposal passed via `ProcessProposal`, even if a part of
the proposal is invalid (e.g., an invalid transaction); the Application can
ignore the invalid part of the prepared proposal at block execution time.
The logic in `ProcessProposal` MUST be deterministic.

- [**ExtendVote:**](./abci++_methods.md#extendvote) It allows applications to force their
validators to do more than just validate within consensus. `ExtendVote` allows applications to
- [**ExtendVote:**](./abci++_methods.md#extendvote) It allows applications to let their
validators do more than just validate within consensus. `ExtendVote` allows applications to
include non-deterministic data, opaque to the consensus algorithm, to precommit messages (the final round of
voting). The data, called *vote extension*, will be broadcast and received together with the
vote it is extending, and will be made available to the Application in the next height,
in the rounds where the local process is the proposer.
CometBFT calls `ExtendVote` when the consensus algorithm is about to send a non-`nil` precommit message.
If the Application does not have vote extension information to provide at that time, it returns
a 0-length byte array as its vote extension.
The logic in `ExtendVote` MAY be non-deterministic.

- [**VerifyVoteExtension:**](./abci++_methods.md#verifyvoteextension) It allows
validators to validate the vote extension data attached to a precommit message. If the validation
Expand All @@ -116,6 +119,7 @@ call sequences of these methods.
As a general rule, an Application that detects an invalid vote extension SHOULD
accept it in `ResponseVerifyVoteExtension` and ignore it in its own logic. CometBFT calls it when
a process receives a precommit message with a (possibly empty) vote extension.
The logic in `VerifyVoteExtension` MUST be deterministic.

- [**FinalizeBlock:**](./abci++_methods.md#finalizeblock) It delivers a decided block to the
Application. The Application must execute the transactions in the block deterministically and
Expand Down Expand Up @@ -202,23 +206,26 @@ More details on managing state across connections can be found in the section on

## Proposal timeout

Immediate execution requires the Application to fully execute the prepared block
before returning from `PrepareProposal`, this means that CometBFT cannot make progress
during the block execution.
This stands on the consensus algorithm critical path: if the Application takes a long time
executing the block, the default value of *TimeoutPropose* might not be sufficient
to accommodate the long block execution time and non-proposer nodes might time
out and prevote `nil`. The proposal, in this case, will probably be rejected and a new round will be necessary.
`PrepareProposal` stands on the consensus algorithm critical path,
i.e., CometBFT cannot make progress while this method is being executed.
Hence, if the Application takes a long time preparing a proposal,
the default value of *TimeoutPropose* might not be sufficient
to accommodate the method's execution and validator nodes might time out and prevote `nil`.
The proposal, in this case, will probably be rejected and a new round will be necessary.


Operators will need to adjust the default value of *TimeoutPropose* in CometBFT's configuration file,
Timeouts are automatically increased for each new round of a height and, if the execution of `PrepareProposal` is bound, eventually *TimeoutPropose* will be long enough to accommodate the execution of `PrepareProposal`.
However, relying on this self adaptation could lead to performance degradation and, therefore,
operators are suggested to adjust the initial value of *TimeoutPropose* in CometBFT's configuration file,
in order to suit the needs of the particular application being deployed.

This is particularly important if applications implement *immediate execution*.
To implement this technique, proposers need to execute the block being proposed within `PrepareProposal`, which could take longer than *TimeoutPropose*.

## Deterministic State-Machine Replication

[↑ Back to Outline](#outline)

ABCI++ applications must implement deterministic finite-state machines to be
ABCI applications must implement deterministic finite-state machines to be
securely replicated by the CometBFT consensus engine. This means block execution
must be strictly deterministic: given the same
ordered set of transactions, all nodes will compute identical responses, for all
Expand All @@ -233,11 +240,13 @@ from block execution (`FinalizeBlock` calls), and not through
any other kind of request. This is the only way to ensure all nodes see the same
transactions and compute the same results.

Some Applications may choose to implement immediate execution, which entails executing the blocks
that are about to be proposed (via `PrepareProposal`), and those that the Application is asked to
validate (via `ProcessProposal`). However, the state changes caused by processing those
Applications that implement immediate execution (execute the blocks
that are about to be proposed, in `PrepareProposal`, or that require validation, in `ProcessProposal`) produce a new candidate state before a block is decided.
The state changes caused by processing those
proposed blocks must never replace the previous state until `FinalizeBlock` confirms
the block decided.
that the proposed block was decided and `Commit` is invoked for it.

The same is true to Applications that quickly accept blocks and execute the blocks optimistically in parallel with the remaining consensus steps to save time during `FinalizeBlock`; they must only apply state changes in `Commit`.

Additionally, vote extensions or the validation thereof (via `ExtendVote` or
`VerifyVoteExtension`) must *never* have side effects on the current state.
Expand Down Expand Up @@ -278,7 +287,7 @@ on them. All other fields in the `Response*` must be strictly deterministic.

Method `FinalizeBlock` includes an `events` field at the top level in its
`Response*`, and one `events` field per transaction included in the block.
Applications may respond to this ABCI++ method with an event list for each executed
Applications may respond to this ABCI 2.0 method with an event list for each executed
transaction, and a general event list for the block itself.
Events allow applications to associate metadata with transactions and blocks.
Events returned via `FinalizeBlock` do not impact the consensus algorithm in any way
Expand Down
46 changes: 26 additions & 20 deletions spec/abci/abci++_comet_expected_behavior.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ Application design should consider _any_ of these possible sequences.

The following grammar, written in case-sensitive Augmented Backus–Naur form (ABNF, specified
in [IETF rfc7405](https://datatracker.ietf.org/doc/html/rfc7405)), specifies all possible
sequences of calls to ABCI++, taken by a correct process, across all heights from the genesis block,
sequences of calls to ABCI++, taken by a **correct process**, across all heights from the genesis block,
including recovery runs, from the point of view of the Application.

```abnf
Expand All @@ -56,7 +56,7 @@ consensus-exec = (inf)consensus-height
consensus-height = *consensus-round decide commit
consensus-round = proposer / non-proposer
proposer = *got-vote [prepare-proposal process-proposal] [extend]
proposer = *got-vote [prepare-proposal [process-proposal]] [extend]
extend = *got-vote extend-vote *got-vote
non-proposer = *got-vote [process-proposal] [extend]
Expand Down Expand Up @@ -130,7 +130,8 @@ Let us now examine the grammar line by line, providing further details.
>```
* In recovery mode, CometBFT first calls `Info` to know from which height it needs to replay decisions
to the Application. After this, CometBFT enters normal consensus execution.
to the Application. After this, CometBFT enters consensus execution, first in replay mode and then
in normal mode.
>```abnf
>recovery = info consensus-exec
Expand All @@ -155,17 +156,26 @@ Let us now examine the grammar line by line, providing further details.
>consensus-round = proposer / non-proposer
>```
* For every round, if the local process is the proposer of the current round, CometBFT calls `PrepareProposal`, followed by `ProcessProposal`.
These two always come together because they reflect the same proposal that the process
also delivers to itself.
* For every round, if the local process is the proposer of the current round, CometBFT calls `PrepareProposal`.
A successful execution of `PrepareProposal` implies in a proposal block being (i)signed and (ii)stored
(e.g., in stable storage).
A crash during this step will direct how the node proceeds the next time it is executed, for the same round, after restarted.
If it crashed before (i), then, during the recovery, `PrepareProposal` will execute as if for the first time.
Following a crash between (i) and (ii) and in (the likely) case `PrepareProposal` produces a different block,
the signing of this block will fail, which means that the new block will not be stored or broadcast.
If the crash happened after (ii), then signing fails but nothing happens to the stored block.
If a block was stored, it is sent to all validators, including the proposer.
Receiving a proposal block triggers `ProcessProposal` with such a block.
Then, optionally, the Application is
asked to extend its vote for that round. Calls to `VerifyVoteExtension` can come at any time: the
local process may be slightly late in the current round, or votes may come from a future round
of this height.
>```abnf
>proposer = *got-vote [prepare-proposal process-proposal] [extend]
>proposer = *got-vote [prepare-proposal [process-proposal]] [extend]
>extend = *got-vote extend-vote *got-vote
>```
Expand Down Expand Up @@ -228,7 +238,7 @@ Finally, `Commit`, which is kept in ABCI++, no longer returns the `AppHash`. It
`FinalizeBlock` to do so. Thus, a slight refactoring of the old `Commit` implementation will be
needed to move the return of `AppHash` to `FinalizeBlock`.
## Accomodating for vote extensions
## Accommodating for vote extensions
In a manner transparent to the application, CometBFT ensures the node is provided with all
the data it needs to participate in consensus.
Expand All @@ -247,22 +257,18 @@ of the usage of `retain_height` stay the same.
The decision to store
historical commits and potential optimizations, are discussed in detail in [RFC-100](./../../docs/rfc/rfc-100-abci-vote-extension-propag.md#current-limitations-and-possible-implementations)
## Handling upgrades to ABCI 2.0
## Handling upgrades to ABCI 2.0
If applications upgrade to ABCI 2.0, CometBFT internally ensures that the [application setup](./abci%2B%2B_app_requirements.md#application-configuration-required-to-switch-to-abci-20) is reflected in its operation.
CometBFT retrieves from the application configuration the value of `VoteExtensionsEnableHeight`( *h<sub>e</sub>*,),
the height at which vote extensions are required for consensus to proceed, and uses it to determine the data it stores and data it sends to a peer
that is catching up.
If applications upgrade to ABCI 2.0, CometBFT internally ensures that the [application setup](./abci%2B%2B_app_requirements.md#application-configuration-required-to-switch-to-abci-20) is reflected in its operation.
CometBFT retrieves from the application configuration the value of `VoteExtensionsEnableHeight`( *h<sub>e</sub>*,),
the height at which vote extensions are required for consensus to proceed, and uses it to determine the data it stores and data it sends to a peer that is catching up.
Namely, upon saving the block for a given height *h* in the block store at decision time
- if *h ≥ h<sub>e</sub>*, the corresponding extended commit that was used to decide locally is saved as well
- if *h < h<sub>e</sub>*, there are no changes to the data saved
* if *h ≥ h<sub>e</sub>*, the corresponding extended commit that was used to decide locally is saved as well
* if *h < h<sub>e</sub>*, there are no changes to the data saved
In the catch-up mechanism, when a node *f* realizes that another peer is at height *h<sub>p</sub>*, which is more than 2 heights behind,
- if *h<sub>p</sub> ≥ h<sub>e</sub>*, *f* uses the extended commit to
* if *h<sub>p</sub> ≥ h<sub>e</sub>*, *f* uses the extended commit to
reconstruct the precommit votes with their corresponding extensions
- if *h<sub>p</sub> < h<sub>e</sub>*, *f* uses the canonical commit to reconstruct the precommit votes,
* if *h<sub>p</sub> < h<sub>e</sub>*, *f* uses the canonical commit to reconstruct the precommit votes,
as done for ABCI 1.0 and earlier.
Loading

0 comments on commit b23ef56

Please sign in to comment.