Add eosio-resume-from-state Test

[kj4ezj]

Change Description

This introduces the eosio-resume-from-state pipeline, which tests that a current version of EOSIO is able to resume from state files generated by previous versions of EOSIO. The versions of EOSIO to test resuming from are declared in the pipeline.jsonc file by git branch, tag, or commit. For example,

{
    "eosio-resume-from-state":
    {
        "test":
        [
            {
                "tag": "v1.8.7"
            },
            {
                "commit": "9bd102691350d1bad49c77e10af21d0474343583" // v2.0.0-rc2
            },
            {
                "branch": "v2.0.0-rc3",
                "tag": "v2.0.0-rc3"
            }
        ]
    }
}

For each test object, the pipeline upload step in eosio-resume-from-state will query for volume snapshots containing state files generated by the given version of nodeos. If it does not find any volume snapshots for any individual test, the pipeline upload step will fail with a useful message so an engineer can review the test and Automation can make the missing snapshot available. Tooling to take these snapshot automatically will be created in the coming weeks. Otherwise, the pipeline upload step finds snapshots for one or more chains and generates a test step for each available chain.

The test steps create volumes from the snapshots and stand up sync nodes using the version of nodeos under test, with no peers. The test passes when the LIB is observed to be greater than or equal to HEIGHT (currently 1 million). The test fails if the LIB does not reflect the state files, nodeos exits, nodeos crashes, or nodeos hangs and the test times out.

The sync nodes and ephemeral volumes are deleted at both the end of the test, and the end of the build. Builds which are cancelled while the sync node is up are the only case where resources are left behind, and we will be creating a reaper in the coming weeks to catch this rare edge case.

Docker Double Delete

This pull request also introduces a fix for the issue we ran into last night with build 19721 trying to delete the same docker container twice.

Tested

• Base Case
  • Build 12 -- eos:develop
  • Build 14 -- eos:release/2.0.x
  • Build 15 -- eos:release/1.8.x
• Edge Cases
  • Build 11 -- various test failures
  • Build 12 -- no snapshot found for one of the tests (see second pipeline upload attempt)

See Also

• auto-eks-sync-nodes Pull Request 11 -- test code
• eos
  • Pull Request 8390 -- develop
  • Pull Request 8391 -- release/2.0.x
  • Pull Request 8392 -- release/1.8.x

Consensus Changes

• Consensus Changes
  None.

API Changes

• API Changes
  None.

Documentation Additions

• Documentation Additions
  None.

[scottarnette]

🍟

[jerstern]