Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

imp(ibc-testkit): Tendermint proof verifications via integration test #1146

Merged
merged 55 commits into from
Apr 5, 2024

Conversation

rnbguy
Copy link
Collaborator

@rnbguy rnbguy commented Mar 29, 2024

Closes: #398

Description

Adds integration tests for combinations of MockHost and TendermintHost.

  • Client creation
  • Connection creation
  • Channel creation
  • Packet relay

Since it includes TendermintHost, these tests also exercise commitment proof validations in Tendermint light clients.

The PR also includes a unit test for minimal proof validations.

Additionally, to maintain the ics23 proof spec, a main_store is introduced in MockGenericContext - that stores the IBC store root at IBC commitment prefix path. The blocks are now generated with the root of the main_store.

To keep the proof retrieval minimal, I had to modify the ProvableContext::get_proof of MockIbcStore. Before it returned the encoded bytes of a singleCommitmentProof. Now it returns the encoded bytes of MerkleProofs, which is essentially a list of CommitmentProofs corresponding to a MerklePath.

Two shortcomings of basecoin_store:


PR author checklist:

  • Added changelog entry, using unclog.
  • Added tests.
  • Linked to GitHub issue.
  • Updated code comments and documentation (e.g., docs/).
  • Tagged one reviewer who will be the one responsible for shepherding this PR.

Reviewer checklist:

  • Reviewed Files changed in the GitHub PR explorer.
  • Manually tested (in case integration/unit/mock tests are absent).

@rnbguy rnbguy force-pushed the rano/testkit/integration-test branch from a7451d3 to 3538f26 Compare March 29, 2024 17:27
Copy link
Member

@Farhad-Shabani Farhad-Shabani left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work on this PR @rnbguy. Adding the commitment proof tests! 🎉
I left a few thoughts in the comments, and as we discussed before we move on:

  • Please review all the newly defined methods and add docstrings.
  • Since we're moving away from the RelayerContext, let's rename and organize TypedIntegration under the relayer directory.

ibc-testkit/src/context.rs Outdated Show resolved Hide resolved
@@ -86,6 +87,8 @@ where
pub packet_ack_store: BinStore<SharedStore<S>, AckPath, AcknowledgementCommitment>,
/// Map of host consensus states
pub host_consensus_states: Arc<Mutex<BTreeMap<u64, AnyConsensusState>>>,
/// Map of older ibc commitment proofs
pub ibc_commiment_proofs: Arc<Mutex<BTreeMap<u64, CommitmentProof>>>,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am thinking our new mock machinery will be going to be an example of IBC module implementation, and even showcasing part of the ibc-testkit in our documentations later. Therefore, I'm unsure about adding this field because commitment proofs usually come from the host storage, not the host's module system, which is the case here too. So, we logically shouldn't need to store it again under the IBC module.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very good point. Actually, I am not sure what proof the IBC gRPC query should return. Is it just the single proof within the IBC store? Is the relayer responsible for getting the proof from the host for the commitment_prefix proof?

I don't see the code for this in ibc-go. So let's clarify this with ibc-go team.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I see. Things differ from ibc-go.
This is a symptom of our QueryContext design. By making the ProverContext the primary trait, we are implicitly requiring the IBC module to be aware of proofs, which logically should not be the case.
Let's proceed for now as I don't see an immediate alternative. However, this issue should be addressed, as it may lead people in the wrong direction.

Copy link
Collaborator Author

@rnbguy rnbguy Apr 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a symptom of our QueryContext design. By making the ProverContext the primary trait, we are implicitly requiring the IBC module to be aware of proofs, which logically should not be the case.

Very good point. I admit that when I implemented this last year, I didn't have this context - so I didn't worry about it.

Carlo's explanation regarding this:

It should include two proofs: one for the IAVL tree of the IBC store (i.e. the proof of inclusion within the IBC store) and another one for Simple Tree (i.e. the proof of inclusion of the IBC store in the multistore).

It seems that the IBC module should be aware of IBC commitment prefix proofs. We should do this like we have host_* methods.

pub trait ProvableContext {
    fn get_ibc_commitment_prefix_proof_on_host(&self, height: Height) -> Option<MerkleProof>
    fn get_local_store_proof(&self, height: Height, path: &Path) -> Option<MerkleProof>;
    fn get_proof(&self, height: Height, path: &Path) -> Option<MerkleProof> {
        let commitment_prefix_proof = self.get_ibc_commitment_prefix_proof_on_host(height);
        let local_proof = get_local_store_proof(height, path);
        return make_merkle_proof_chain([commitment_prefix_proof, local_proof]);
    }
}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed, the architecture of Cosmos SDK chains may not directly apply to other non-Cosmos chains or rollups. Let's explore this further as a separate issue. Refactoring these parts would likely require a bit more studying and experimentation.

Comment on lines 126 to 132
fn header_params<C>(
&self,
client_id: &ClientId,
client_context: &C,
) -> HostLightClientHeaderParams<Self>
where
C: ClientValidationContext;
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Introduced this to generalize the parameters needed to convert a block to a light client header.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be honest, I found this new method a bit confusing and difficult to grasp its purpose. Since we have tight plans, I made some simplifications (410b8ad) so that can move forward and wrap up ibc-testkit refactoring. I hope that works for you!

@Farhad-Shabani Farhad-Shabani merged commit 2acb2fa into feat/refactor-testkit Apr 5, 2024
16 checks passed
@Farhad-Shabani Farhad-Shabani deleted the rano/testkit/integration-test branch April 5, 2024 02:17
github-merge-queue bot pushed a commit that referenced this pull request May 3, 2024
* imp(testkit): Mock IBC context with `basecoin-store` types  (#1068)

* non empty default CommitmentPrefix

* update ibc mock client types

* fix tests for updated mock types

* generalize MockContextConfig::into

* rm clone from MockContext

* add with_{client,consensus}_state

* add blocks_since in utils.rs

* client takes host timestamp by default

* refactor relayer context test

* fix few tests

* add ibc-query and basecoin-store deps

* add basecoin-store in mock ibc context

* refactor for updated mock ibc context

* imp timeout test

* public git commit as dep source

* fix spelling

* update MockClient types

* fix failing tests

* rm unused utils and deps

* fix cargo doc lint

* use ibc host paths in mock context

* rm redundant curly brackets

---------

Co-authored-by: Farhad Shabani <farhad.shabani@gmail.com>

* imp(ibc-testkit): generalize `Host` for `MockContext` (#1107)

* fix semantic conflict

* happy clippy

* fix next_consensus_state impl

* update prev_consensus_state impl

* use self.block_time over const

* use constant timestamp over dynamic now

* add new host impls

* changes to ibc-testkit

* rm deprecated methods

* refactor tests

* public fields for tm block params

* forged client header update test

* add todo comment

* minor refactor

* refactor tests

* builder type for light client state

* minor refactor

* refactor tests

* code opt

* rename host structs

* renamings

* refactor tendermint host

* mv year_2023 to utils

* refactor the new tests

* test malicious adjacent header update

* fix tests

* add changelog

* rm unused chain_revision_number

* add doc string for year_2023

* rename to query_latest_block

* feat(ibc-testkit): revamp `MockContext` with chain capabilities (#1135)

* imp: impl ctxs on MockIbcStore

* fix: set update_meta whenever build light client

* imp: generate_client_state should take latest_height

* imp: remove unnecessary latest_client_states method

* refactor: migrate host relevant fields/methods

* fix: get back validate_self_client

* imp: bring context types up under src

* rm default for CommitmentPrefix

* rm default height and timestamp impl for TestBlock

* default MockStore

* AnyClient and AnyConsensus state in MockIbcStore

* at least one consensus state when bootstrapping light client

* revision number in MockIbcStore

* advance height in MockIbcStore

* sync host and ibc store advance in conext

* add host params to build host

* update host trait

* convenient type generics for host associated types

* update MockHost

* return existing block header with correct timestamp

* update TendermintHost

* update MockGenericContext impl

* update mock context building

* add implied trait bounds

* refactor method

* rm redundant imports and impl

* call advance_block on context

* update MockContext tests

* rm ClientStateCommon

* use HostParams to build a host

* rm using max_history_size

* update few tests

* ignore failing tests

* clippy::use_self

* clippy::flat_map_option

* clippy::cloned_instead_of_copied

* clippy::redundant_clone

* clippy::redundant_type_annotations

* clippy::as_underscore

* disable slow testcase

* prune old host consensus state

* prune host consensus history in fixture

* enable ignored test

* return client_id in fixture

* fix and enable failing test

* avoid Arc and Mutex

* fix doc build

* refactor host trait and impls

* into over Self::from

* fix msrv

* explicit relative or global path for pub use

* clippy::map_identity

* clippy::inconsistent_struct_constructor

* clippy::std_instead_of_core

* use commitment_root for block generation

* rename consensus_states to host_consensus_states

* use basecoin proof specs

* update tests

* imp: add history() under TestHost

* chore: add docstring for some of methods under TestHost

* update TestHost trait

* update MockHost and TendermintHost

* add MockGenericContext::generate_genesis_block

* update MockContextConfig

* update tests

* update remaining tests

* rm comments and rename test

* rm HostParams

---------

Co-authored-by: Farhad Shabani <farhad.shabani@gmail.com>

* imp(ibc-testkit): Tendermint proof verifications via integration test (#1146)

* test tmclient proof verification

* use ClientStateConfig directly

* rm MockClientConfig

* use basecoin store proofspec as default

* update tests

* use merkle storage in MockContext

* fix ProofSpecs size

* refactor MockIbcStore to perform begin_block and end_block

* simpler proof verification test

* use ValidationContext::commitment_prefix()

* nits

* refactor host related trait method

* tendermint host client integration test with proof verification

* rm raw test

* use typed relayer

* add todo for channel integration test

* core over std

* be semantically correct

* add comment for TypedRelayer

* integration test for all pairs

* fix semantic bug

* renames

* add channel management

* channel creation in RelayerContext

* add channel creation in integration test

* add test for channel close

* query client_id from connection and channel

* ibc_store_mut

* utils functions for packet relay

* add packet relay integration test

* add comments

* optimize integration utils functions

* serde feature for integration tests

* rm redundant chain_id

* sync clock only on a

* add comment

* imp: place router under MockGenericContext

* nit: add docstring for router

* nits

* rm redundant lint filters

* imp: ditch RelayerContext

* nit: simplify build_client_update_datagram

* refactor integration tests

* add doc strings for TypedRelayerOps

* doc strings for RelayerContext

* mv client_update_ping_pong to tests dir

* rename main_store to multi_store

* update TestHost trait

* update mock and tendermint hosts

* update relayer functions

* nits

* renames and comments

* add comments for return values in relayer ops

* imp: simplify into_header

---------

Co-authored-by: Farhad Shabani <farhad.shabani@gmail.com>

* rm prune_block_till

* apply suggestions from code review

Co-authored-by: Sean Chen <seanchen11235@gmail.com>
Signed-off-by: Rano | Ranadeep <ranadip.bswas@gmail.com>

* rename context structs

* add MockContext and TendermintContext

* update with MockContext and TendermintContext

* MockContextConfig to TestContextConfig

* update comments with new namings

* add util methods for StoreGenericTestContext

* fix failing client recovery tests

* increase default trusting period of MockClientState

* fallible ProofSpecs conversion

* update with new ClientStateConfig

* use ExtClientValidationContext

* fix links in doc comments

* fix import order

* fix semantic conflicts

* Convertible over ConsensusStateConverter

* rm ConsensusStateConverter

* ibc-query as workspace deps

* update Convertible

* fix failing test

* update changelog entry

* add comments

* rm absolute import

* use expect over comment for safety

* infallible try_into

* update method names

* add unused variable names

* rm redundant TendermintBlock wrapper type

* add comments

* update docstrings

* rm should never fail comment

* update adr

* fix compilation

* fix failing test

* docs: ADR-009 to revamp testing framework (#1157)

* draft adr 009

* update adr-009

* apply suggestions from pr review

Co-authored-by: Sean Chen <seanchen11235@gmail.com>
Signed-off-by: Rano | Ranadeep <ranadip.bswas@gmail.com>

* use numbers list for sub-proposal titles

Co-authored-by: Sean Chen <seanchen11235@gmail.com>
Signed-off-by: Rano | Ranadeep <ranadip.bswas@gmail.com>

* links to validation and execute context

Co-authored-by: Sean Chen <seanchen11235@gmail.com>
Signed-off-by: Rano | Ranadeep <ranadip.bswas@gmail.com>

* apply suggestion from pr review

* add links to impls

* update comment

* markdown format

* update adr

* use TestContext

* update comment

* apply suggestions from code review

Co-authored-by: Sean Chen <seanchen11235@gmail.com>
Signed-off-by: Rano | Ranadeep <ranadip.bswas@gmail.com>

* markdown format

* update method names

* fix cargo doc error

---------

Signed-off-by: Rano | Ranadeep <ranadip.bswas@gmail.com>
Co-authored-by: Sean Chen <seanchen11235@gmail.com>

---------

Signed-off-by: Rano | Ranadeep <ranadip.bswas@gmail.com>
Signed-off-by: Rano | Ranadeep <ranadeep@informal.systems>
Co-authored-by: Farhad Shabani <farhad.shabani@gmail.com>
Co-authored-by: Sean Chen <seanchen11235@gmail.com>
Farhad-Shabani added a commit that referenced this pull request Sep 9, 2024
* imp(testkit): Mock IBC context with `basecoin-store` types  (#1068)

* non empty default CommitmentPrefix

* update ibc mock client types

* fix tests for updated mock types

* generalize MockContextConfig::into

* rm clone from MockContext

* add with_{client,consensus}_state

* add blocks_since in utils.rs

* client takes host timestamp by default

* refactor relayer context test

* fix few tests

* add ibc-query and basecoin-store deps

* add basecoin-store in mock ibc context

* refactor for updated mock ibc context

* imp timeout test

* public git commit as dep source

* fix spelling

* update MockClient types

* fix failing tests

* rm unused utils and deps

* fix cargo doc lint

* use ibc host paths in mock context

* rm redundant curly brackets

---------

Co-authored-by: Farhad Shabani <farhad.shabani@gmail.com>

* imp(ibc-testkit): generalize `Host` for `MockContext` (#1107)

* fix semantic conflict

* happy clippy

* fix next_consensus_state impl

* update prev_consensus_state impl

* use self.block_time over const

* use constant timestamp over dynamic now

* add new host impls

* changes to ibc-testkit

* rm deprecated methods

* refactor tests

* public fields for tm block params

* forged client header update test

* add todo comment

* minor refactor

* refactor tests

* builder type for light client state

* minor refactor

* refactor tests

* code opt

* rename host structs

* renamings

* refactor tendermint host

* mv year_2023 to utils

* refactor the new tests

* test malicious adjacent header update

* fix tests

* add changelog

* rm unused chain_revision_number

* add doc string for year_2023

* rename to query_latest_block

* feat(ibc-testkit): revamp `MockContext` with chain capabilities (#1135)

* imp: impl ctxs on MockIbcStore

* fix: set update_meta whenever build light client

* imp: generate_client_state should take latest_height

* imp: remove unnecessary latest_client_states method

* refactor: migrate host relevant fields/methods

* fix: get back validate_self_client

* imp: bring context types up under src

* rm default for CommitmentPrefix

* rm default height and timestamp impl for TestBlock

* default MockStore

* AnyClient and AnyConsensus state in MockIbcStore

* at least one consensus state when bootstrapping light client

* revision number in MockIbcStore

* advance height in MockIbcStore

* sync host and ibc store advance in conext

* add host params to build host

* update host trait

* convenient type generics for host associated types

* update MockHost

* return existing block header with correct timestamp

* update TendermintHost

* update MockGenericContext impl

* update mock context building

* add implied trait bounds

* refactor method

* rm redundant imports and impl

* call advance_block on context

* update MockContext tests

* rm ClientStateCommon

* use HostParams to build a host

* rm using max_history_size

* update few tests

* ignore failing tests

* clippy::use_self

* clippy::flat_map_option

* clippy::cloned_instead_of_copied

* clippy::redundant_clone

* clippy::redundant_type_annotations

* clippy::as_underscore

* disable slow testcase

* prune old host consensus state

* prune host consensus history in fixture

* enable ignored test

* return client_id in fixture

* fix and enable failing test

* avoid Arc and Mutex

* fix doc build

* refactor host trait and impls

* into over Self::from

* fix msrv

* explicit relative or global path for pub use

* clippy::map_identity

* clippy::inconsistent_struct_constructor

* clippy::std_instead_of_core

* use commitment_root for block generation

* rename consensus_states to host_consensus_states

* use basecoin proof specs

* update tests

* imp: add history() under TestHost

* chore: add docstring for some of methods under TestHost

* update TestHost trait

* update MockHost and TendermintHost

* add MockGenericContext::generate_genesis_block

* update MockContextConfig

* update tests

* update remaining tests

* rm comments and rename test

* rm HostParams

---------

Co-authored-by: Farhad Shabani <farhad.shabani@gmail.com>

* imp(ibc-testkit): Tendermint proof verifications via integration test (#1146)

* test tmclient proof verification

* use ClientStateConfig directly

* rm MockClientConfig

* use basecoin store proofspec as default

* update tests

* use merkle storage in MockContext

* fix ProofSpecs size

* refactor MockIbcStore to perform begin_block and end_block

* simpler proof verification test

* use ValidationContext::commitment_prefix()

* nits

* refactor host related trait method

* tendermint host client integration test with proof verification

* rm raw test

* use typed relayer

* add todo for channel integration test

* core over std

* be semantically correct

* add comment for TypedRelayer

* integration test for all pairs

* fix semantic bug

* renames

* add channel management

* channel creation in RelayerContext

* add channel creation in integration test

* add test for channel close

* query client_id from connection and channel

* ibc_store_mut

* utils functions for packet relay

* add packet relay integration test

* add comments

* optimize integration utils functions

* serde feature for integration tests

* rm redundant chain_id

* sync clock only on a

* add comment

* imp: place router under MockGenericContext

* nit: add docstring for router

* nits

* rm redundant lint filters

* imp: ditch RelayerContext

* nit: simplify build_client_update_datagram

* refactor integration tests

* add doc strings for TypedRelayerOps

* doc strings for RelayerContext

* mv client_update_ping_pong to tests dir

* rename main_store to multi_store

* update TestHost trait

* update mock and tendermint hosts

* update relayer functions

* nits

* renames and comments

* add comments for return values in relayer ops

* imp: simplify into_header

---------

Co-authored-by: Farhad Shabani <farhad.shabani@gmail.com>

* rm prune_block_till

* apply suggestions from code review

Co-authored-by: Sean Chen <seanchen11235@gmail.com>
Signed-off-by: Rano | Ranadeep <ranadip.bswas@gmail.com>

* rename context structs

* add MockContext and TendermintContext

* update with MockContext and TendermintContext

* MockContextConfig to TestContextConfig

* update comments with new namings

* add util methods for StoreGenericTestContext

* fix failing client recovery tests

* increase default trusting period of MockClientState

* fallible ProofSpecs conversion

* update with new ClientStateConfig

* use ExtClientValidationContext

* fix links in doc comments

* fix import order

* fix semantic conflicts

* Convertible over ConsensusStateConverter

* rm ConsensusStateConverter

* ibc-query as workspace deps

* update Convertible

* fix failing test

* update changelog entry

* add comments

* rm absolute import

* use expect over comment for safety

* infallible try_into

* update method names

* add unused variable names

* rm redundant TendermintBlock wrapper type

* add comments

* update docstrings

* rm should never fail comment

* update adr

* fix compilation

* fix failing test

* docs: ADR-009 to revamp testing framework (#1157)

* draft adr 009

* update adr-009

* apply suggestions from pr review

Co-authored-by: Sean Chen <seanchen11235@gmail.com>
Signed-off-by: Rano | Ranadeep <ranadip.bswas@gmail.com>

* use numbers list for sub-proposal titles

Co-authored-by: Sean Chen <seanchen11235@gmail.com>
Signed-off-by: Rano | Ranadeep <ranadip.bswas@gmail.com>

* links to validation and execute context

Co-authored-by: Sean Chen <seanchen11235@gmail.com>
Signed-off-by: Rano | Ranadeep <ranadip.bswas@gmail.com>

* apply suggestion from pr review

* add links to impls

* update comment

* markdown format

* update adr

* use TestContext

* update comment

* apply suggestions from code review

Co-authored-by: Sean Chen <seanchen11235@gmail.com>
Signed-off-by: Rano | Ranadeep <ranadip.bswas@gmail.com>

* markdown format

* update method names

* fix cargo doc error

---------

Signed-off-by: Rano | Ranadeep <ranadip.bswas@gmail.com>
Co-authored-by: Sean Chen <seanchen11235@gmail.com>

---------

Signed-off-by: Rano | Ranadeep <ranadip.bswas@gmail.com>
Signed-off-by: Rano | Ranadeep <ranadeep@informal.systems>
Co-authored-by: Farhad Shabani <farhad.shabani@gmail.com>
Co-authored-by: Sean Chen <seanchen11235@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

2 participants