EPIC: Integration test framework

[tac0turtle]

Summary

All forms of testing it important when developing an application. Secondly, having a testing framework for different levels of testing (unit, integration, e2e) provides a seamless way for users to write clean and reliable tests.

Currently, there is not an integration framework available.

Problem Definition

To write an integration test suite for a module, users must build an application or use their app.go for testing. This creates a test environment that is in-between an e2e test and integration test. Secondly, the e2e tests are not well-defined and require tendermint on top of modules that are not needed in a modules testing (staking, auth, bank, params, consensus, etc..)

Work Breakdown

Design and implement an integration framework for applications that only requires the module and its dependencies to be included in the dependency graph. This could be something as simple as a few routers for queries, messages, and anything else that is needed, but not an entire application that requires things like staking if it's not required.

Secondly, this integration framework could later be used for benchmarking specific logic of a module without the need to spin up an application.

Preliminary phase

• Replace usage of testify suite in integration tests: Replace testify usage by gotest.tools #12332 (this issue should not be closed until the whole SDK is migrated)
  • Replace testify with gotest.tools in integration/genutil tests #14492
  • Replace testify with gotest.tools in in integration/distribution tests #14494
  • Replace testify with gotest.tools in integration/store tests #14496
  • Replace testify with gotest.tools in integration/gov tests #14500
  • Replace testify with gotest.tools in integration/bank tests #14493
  • evidence
  • runtime
  • Replace testify with gotest.tools in integration/slashing tests #14498
  • Replace testify with gotest.tools in integration/staking tests #14503

Phase 1

• Research an integration framework and create PoC #14676
• Migrate tests/integration to demonstrate
  • Migrate x/distribution to use integration test framework #14678
  • Migrate x/staking to use integration test framework #14677
• Write documentation on how to use it
  • Update https://docs.cosmos.network/main/architecture/adr-059-test-scopes.html as well.
  • Update https://docs.cosmos.network/main/building-modules/testing

Phase 2

• Migrate modules to use integration tests helpers
  • bank test(x/bank): write integration tests #16052
  • distribution test(x/distribution): write integration tests #15569
  • evidence test(x/evidence): write integration tests #16015
  • gov test(x/gov): write integration tests #16194
  • genutil test(genutil): remove duplicate tests from genutil integration tests #16034
  • slashing test(x/slashing): write integration tests #15928
  • staking test(x/staking): write integration tests #15890
• Refactor tests/integrations for all modules
• Verify test duplication between new integration tests, CLI and e2e tests. Remove the duplicated tests from e2e when applicable.
  • Verify for all modules with integration tests
  • auth
  • authz
  • bank
  • crisis
  • distribution
  • feegrant
  • gov
  • group
  • mint
  • nft
  • params
  • staking
  • tx
  • upgrade

[tac0turtle]

double checking on this epic, is it up to date with the latest changes?

[likhita-809]

yes, for the remaining modules either tests are not verified for duplication / partially deduplicated tests.
So we just need to verify duplicate tests btw integration, CLI and e2e tests in the remaining modules.

[tac0turtle]

closing this and opening a new epic around dedupication of e2e/cli/integration tests