CI: Minor releases should sync through more blocks pre-release

[ValarDragon]

Background

Currently we have a CI workflow that runs epoch block + 1000 blocks on every commit to latest release branch. This is great for testing!

We should also have a CI flow that we can use pre-release that syncs through every available block on that branch. (So we make a snapshot of state at a height slightly after upgrade or upgrade directly)

This will increase our pre-release mainnet sync's battery of tests, from 1000 blocks (+ simulator after #2267 ) to all available blocks. The degree to which we can depend on this will depend on whats the syncing advantage factor, what degree we can parallelize the syncs, etc.

Suggested Design

Make a CI workflow we can use to try syncing a node with a given branch, from pre-configured start height for that release (e.g. first block of v11 based on a snapshot) to some set end block OR latest block.

This can be something we manually require before tagging any minor or patch version

[niccoloraspa]

Hey @ValarDragon, thanks for the issue. I like the idea of running the check for more extended periods before a release.

I had a more basic approach to addressing this problem:
What I had in mind was to have a CI that would spin up a node from the latest snapshot and leave it up for a certain amount of time (e.g., 24h hours), setting a halt-time in the config.toml.

This is simpler to implement as it is only a tiny variation of our current CI, and we would not need to have a post-upgrade snapshot. At the same time:

• it would only process a portion of the blocks and not all the vX blocks
• it would "freeze" the release for the testing time

What I like about this though, it's that the CI offers node-as-service and it can be used to test any branch/commit for longer period of time and is not necessarily limited to a release.

If it's okay with you, we can start with this solution first and then modifying to begin at earlier block heights.
The main reason for adding this intermediate step it's that we would need to extend our self-hosted architecture to allow a node to run for an extended period of time.

[faddat]

#3507

Does anyone other than myself have a state sync script?

I favor Dev's approach, and use it widely on other chains. It is a safe way to check for app hash errors. It requires linking a physical machine into ci because otherwise it takes forever and a day.

[czarcas7ic]

@niccoloraspa is there any updates on the path we are looking to go with this?

[niccoloraspa]

I have moved this task to our board.
We could close this issue here.

[niccoloraspa]

It's tracked here for reference: https://github.com/osmosis-labs/osmosis-ci/issues/13