Chain-id upgrade fails upgrading IBC client in remote chain

[RaulBernal]

Summary of Bug

Following the instructions at https://hermes.informal.systems/documentation/commands/upgrade/test.html I've found an issue when Hermes should waits for the reference chain bitcanna-dev-1 (ibc-0 in the example) to halt and then performs the upgrade for client 07-tendermint-0 on osmosis-dev-1 (ibc-1 at the Hermes doc)

Version

$hermes version
2023-02-02T15:43:55.032433Z  INFO ThreadId(01) running Hermes v1.2.0
hermes 1.2.0

I run it in Darwin arm64 M1

arch -arm64 brew install rust
==> Installing rust
==> Pouring rust--1.67.0.arm64_monterey.bottle.tar.gz
==> Caveats
cargo install ibc-relayer-cli --bin hermes --locked
Installed package `ibc-relayer-cli v1.2.0` (executable `hermes`)

Steps to Reproduce

Run two chains and connect it by IBC; make some IBC TX in both senses, relaying them with Hermes.
Version of chain's daemons are:
Osmosis:

name: osmosis
server_name: osmosisd
version: 14.0.0-rc1
commit: 26e2fad8e7b3eb7c33965360b31a593b392d7d75
build_tags: netgo,ledger
go: go version go1.19.5 darwin/amd64

BitCanna:

commit: c9bcec3da8fb49a11981561ba3f539bc21cf0bc6
cosmos_sdk_version: v0.46.8
go: go version go1.19.5 darwin/amd64
name: Bcna
server_name: bcnad
version: 2.0.1-rc4

I want upgrade bitcanna-dev-1 to bitcanna-dev-2 keeping 07-tendermint-0 working on both chains (same ClientID in both)

I make the IBC upgrade proposal:

$ hermes tx upgrade-chain --reference-chain bitcanna-dev-1 --new-chain bitcanna-dev-2 --denom ubcna --host-chain osmosis-dev-1 --host-client 07-tendermint-0 --amount 10000000 --height-offset 250 --upgrade-name change_chain_id 
2023-02-02T08:03:36.283954Z  INFO ThreadId(01) using default configuration from '/Users/raul/.hermes/config.toml'
2023-02-02T08:03:36.284111Z  INFO ThreadId(01) running Hermes v1.2.0
SUCCESS Hash::Sha256(92377AEC8E24E7334DF1F2D1A6F7B3D5E9AB5B3BDB8F7FB8C4AD914B3D4611C5) 

Proposal 2 is created & passed: Halt-height is 2705

$ bcnad query gov proposals
id: "2"
  messages:
  - '@type': /cosmos.gov.v1.MsgExecLegacyContent
    authority: bcna10d07y265gmmuvt4z0w9aw880jnsr700jqjry7p
    content:
      '@type': /ibc.core.client.v1.UpgradeProposal
      description: upgrade the chain software and unbonding period
      plan:
        height: "2705"
        info: ""
        name: change_chain_id
        time: "0001-01-01T00:00:00Z"
        upgraded_client_state: null
      title: proposal 0
      upgraded_client_state:
        '@type': /ibc.lightclients.tendermint.v1.ClientState
        allow_update_after_expiry: false
        allow_update_after_misbehaviour: false
        chain_id: bitcanna-dev-2
        frozen_height:
          revision_height: "0"
          revision_number: "0"
        latest_height:
          revision_height: "2706"
          revision_number: "1"
        max_clock_drift: 0s
        proof_specs:
        - inner_spec:
            child_order:
            - 0
            - 1
            child_size: 33
            empty_child: null
            hash: SHA256
            max_prefix_length: 12
            min_prefix_length: 4
          leaf_spec:
            hash: SHA256
            length: VAR_PROTO
            prefix: AA==
            prehash_key: NO_HASH
            prehash_value: SHA256
          max_depth: 0
          min_depth: 0
        - inner_spec:
            child_order:
            - 0
            - 1
            child_size: 32
            empty_child: null
            hash: SHA256
            max_prefix_length: 1
            min_prefix_length: 1
          leaf_spec:
            hash: SHA256
            length: VAR_PROTO
            prefix: AA==
            prehash_key: NO_HASH
            prehash_value: SHA256
          max_depth: 0
          min_depth: 0
        trust_level:
          denominator: "0"
          numerator: "0"
        trusting_period: 0s
        unbonding_period: 1814400s
        upgrade_path:
        - upgrade
        - upgradedIBCState
  metadata: ""
  status: PROPOSAL_STATUS_VOTING_PERIOD
  submit_time: "2023-02-02T08:03:34.886593Z"
  total_deposit:
  - amount: "10000000"
    denom: ubcna
  voting_end_time: "2023-02-02T08:08:34.886593Z"
  voting_start_time: "2023-02-02T08:03:34.886593Z"

Then I start the Hermes handler (before halt-height) (tested also passed the halt-height):
$ hermes upgrade client --host-chain osmosis-dev-1 --client 07-tendermint-0 --upgrade-height 2705
Output when the halt-height is reached:

2023-02-02T08:05:11.714513Z  INFO ThreadId(01) running Hermes v1.2.0
2023-02-02T08:24:52.742879Z ERROR ThreadId(11) foreign_client.upgrade{src_upgrade_height=Height { revision: 1, height: 2705 } client=bitcanna-dev-1->osmosis-dev-1:07-tendermint-0}:send_messages_and_wait_commit{chain=osmosis-dev-1 tracking_id=upgrade client}:send_tx_with_account_sequence_retry{chain=osmosis-dev-1 account.sequence=11}:estimate_gas: failed to simulate tx. propagating error to caller: gRPC call failed with status: status: Unknown, message: "failed to execute message; message index: 1: cannot upgrade client with ID 07-tendermint-0: updated client state failed basic validation: latest height revision number must match chain id revision number (1 != 2): invalid header height", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "773"} }
2023-02-02T08:24:52.742942Z ERROR ThreadId(11) foreign_client.upgrade{src_upgrade_height=Height { revision: 1, height: 2705 } client=bitcanna-dev-1->osmosis-dev-1:07-tendermint-0}:send_messages_and_wait_commit{chain=osmosis-dev-1 tracking_id=upgrade client}:send_tx_with_account_sequence_retry{chain=osmosis-dev-1 account.sequence=11}: gas estimation failed or encountered another unrecoverable error error=gRPC call failed with status: status: Unknown, message: "failed to execute message; message index: 1: cannot upgrade client with ID 07-tendermint-0: updated client state failed basic validation: latest height revision number must match chain id revision number (1 != 2): invalid header height", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "773"} }
ERROR failed while trying to upgrade client id 07-tendermint-0 for chain osmosis-dev-1: failed while sending message to destination chain: gRPC call failed with status: status: Unknown, message: "failed to execute message; message index: 1: cannot upgrade client with ID 07-tendermint-0: updated client state failed basic validation: latest height revision number must match chain id revision number (1 != 2): invalid header height", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "773"} }: gRPC call failed with status: status: Unknown, message: "failed to execute message; message index: 1: cannot upgrade client with ID 07-tendermint-0: updated client state failed basic validation: latest height revision number must match chain id revision number (1 != 2): invalid header height", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "773"} } 

At Osmosis-dev-1:

9:24AM INF indexed block exents height=773 module=txindex
9:24AM INF client state updated client-id=07-tendermint-0 height=1-2705 module=x/ibc/client 

Really is updated in the host chain but not changed to revision 2 (bitcanna-dev-2) (the desired effect)
The first idea it comes to my mind is that issue is here: latest height revision number must match chain id revision number (1 != 2) .

Acceptance Criteria

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate milestone (priority) applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[adizere]

I managed to reproduce this. Thank you @RaulBernal for submitting the issue. It's not clear what the problem is. Your reproduction steps are very thorough, so I won't cover that. I'll just paste here the detailed log on how the failure looks like:

% hermes upgrade client --host-chain hu-3 --client 07-tendermint-0 --upgrade-height 488
2023-02-02T17:28:31.626820Z  INFO ThreadId(01) using default configuration from '/Users/adiseredinschi/.hermes/config.toml'
2023-02-02T17:28:31.627918Z  INFO ThreadId(01) running Hermes v1.2.0+097c5dfd
2023-02-02T17:28:31.644437Z DEBUG ThreadId(01) Reference application latest height: 5-487
2023-02-02T17:28:37.653003Z TRACE ThreadId(29) foreign_client.upgrade{src_upgrade_height=Height { revision: 5, height: 488 } client=oz-5->hu-3:07-tendermint-0}:foreign_client.build_update_client_with_trusted{client=oz-5->hu-3:07-tendermint-0 target_height=5-488}: light client verification trusted=5-27 target=5-488
2023-02-02T17:28:37.669707Z TRACE ThreadId(29) foreign_client.upgrade{src_upgrade_height=Height { revision: 5, height: 488 } client=oz-5->hu-3:07-tendermint-0}:foreign_client.build_update_client_with_trusted{client=oz-5->hu-3:07-tendermint-0 target_height=5-488}: adjusting headers with 0 supporting headers trusted=5-27 target=488
2023-02-02T17:28:37.669777Z TRACE ThreadId(29) foreign_client.upgrade{src_upgrade_height=Height { revision: 5, height: 488 } client=oz-5->hu-3:07-tendermint-0}:foreign_client.build_update_client_with_trusted{client=oz-5->hu-3:07-tendermint-0 target_height=5-488}: fetching header height=5-28
2023-02-02T17:28:37.679101Z DEBUG ThreadId(01) foreign_client.upgrade{src_upgrade_height=Height { revision: 5, height: 488 } client=oz-5->hu-3:07-tendermint-0}:foreign_client.build_update_client_with_trusted{client=oz-5->hu-3:07-tendermint-0 target_height=5-488}: building a MsgUpdateAnyClient from trusted height 5-27 to target height 5-488
2023-02-02T17:28:37.680867Z DEBUG ThreadId(01) foreign_client.upgrade{src_upgrade_height=Height { revision: 5, height: 488 } client=oz-5->hu-3:07-tendermint-0}: upgraded client state Tendermint(ClientState { chain_id: ChainId { id: "oz-6", version: 6 }, trust_threshold: TrustThreshold(Ratio { numer: 0, denom: 0 }), trusting_period: 0ns, unbonding_period: 1814400s, max_clock_drift: 0ns, latest_height: Height { revision: 5, height: 489 }, proof_specs: ProofSpecs([ProofSpec(ProofSpec { leaf_spec: Some(LeafOp { hash: Sha256, prehash_key: NoHash, prehash_value: Sha256, length: VarProto, prefix: [0] }), inner_spec: Some(InnerSpec { child_order: [0, 1], child_size: 33, min_prefix_length: 4, max_prefix_length: 12, empty_child: [], hash: Sha256 }), max_depth: 0, min_depth: 0 }), ProofSpec(ProofSpec { leaf_spec: Some(LeafOp { hash: Sha256, prehash_key: NoHash, prehash_value: Sha256, length: VarProto, prefix: [0] }), inner_spec: Some(InnerSpec { child_order: [0, 1], child_size: 32, min_prefix_length: 1, max_prefix_length: 1, empty_child: [], hash: Sha256 }), max_depth: 0, min_depth: 0 })]), upgrade_path: ["upgrade", "upgradedIBCState"], allow_update: AllowUpdate { after_expiry: false, after_misbehaviour: false }, frozen_height: None, verifier: PredicateVerifier { predicates: ProdPredicates, voting_power_calculator: ProdVotingPowerCalculator, commit_validator: ProdCommitValidator { hasher: ProdHasher }, hasher: ProdHasher } })
2023-02-02T17:28:37.683377Z DEBUG ThreadId(01) foreign_client.upgrade{src_upgrade_height=Height { revision: 5, height: 488 } client=oz-5->hu-3:07-tendermint-0}: upgraded client consensus state Tendermint(ConsensusState { timestamp: Time(2023-02-02 17:15:20.746298), root: CommitmentRoot(""), next_validators_hash: Hash::Sha256(125DAD56E6A45ED1B7F870BF6AE5F7ADF975FC0304742A76E037D0A9EB1EE566) })
2023-02-02T17:28:37.690130Z DEBUG ThreadId(11) foreign_client.upgrade{src_upgrade_height=Height { revision: 5, height: 488 } client=oz-5->hu-3:07-tendermint-0}:send_messages_and_wait_commit{chain=hu-3 tracking_id=upgrade client}: sending 2 messages as 1 batches to chain hu-3 in parallel
2023-02-02T17:28:37.690275Z DEBUG ThreadId(11) foreign_client.upgrade{src_upgrade_height=Height { revision: 5, height: 488 } client=oz-5->hu-3:07-tendermint-0}:send_messages_and_wait_commit{chain=hu-3 tracking_id=upgrade client}:send_tx_with_account_sequence_retry{chain=hu-3 account.sequence=1}: max fee, for use in tx simulation: Fee { amount: "100001stake", gas_limit: 10000000 }
2023-02-02T17:28:37.696046Z ERROR ThreadId(11) foreign_client.upgrade{src_upgrade_height=Height { revision: 5, height: 488 } client=oz-5->hu-3:07-tendermint-0}:send_messages_and_wait_commit{chain=hu-3 tracking_id=upgrade client}:send_tx_with_account_sequence_retry{chain=hu-3 account.sequence=1}:estimate_gas: failed to simulate tx. propagating error to caller: gRPC call failed with status: status: Unknown, message: "failed to execute message; message index: 1: cannot upgrade client with ID 07-tendermint-0: updated client state failed basic validation: latest height revision number must match chain id revision number (5 != 6): invalid header height [cosmos/ibc-go/v3@v3.0.0/modules/light-clients/07-tendermint/types/client_state.go:130] With gas wanted: '0' and gas used: '120739' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "641"} }
2023-02-02T17:28:37.696216Z ERROR ThreadId(11) foreign_client.upgrade{src_upgrade_height=Height { revision: 5, height: 488 } client=oz-5->hu-3:07-tendermint-0}:send_messages_and_wait_commit{chain=hu-3 tracking_id=upgrade client}:send_tx_with_account_sequence_retry{chain=hu-3 account.sequence=1}: gas estimation failed or encountered another unrecoverable error error=gRPC call failed with status: status: Unknown, message: "failed to execute message; message index: 1: cannot upgrade client with ID 07-tendermint-0: updated client state failed basic validation: latest height revision number must match chain id revision number (5 != 6): invalid header height [cosmos/ibc-go/v3@v3.0.0/modules/light-clients/07-tendermint/types/client_state.go:130] With gas wanted: '0' and gas used: '120739' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "641"} }
ERROR failed while trying to upgrade client id 07-tendermint-0 for chain hu-3: failed while sending message to destination chain: gRPC call failed with status: status: Unknown, message: "failed to execute message; message index: 1: cannot upgrade client with ID 07-tendermint-0: updated client state failed basic validation: latest height revision number must match chain id revision number (5 != 6): invalid header height [cosmos/ibc-go/v3@v3.0.0/modules/light-clients/07-tendermint/types/client_state.go:130] With gas wanted: '0' and gas used: '120739' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "641"} }: gRPC call failed with status: status: Unknown, message: "failed to execute message; message index: 1: cannot upgrade client with ID 07-tendermint-0: updated client state failed basic validation: latest height revision number must match chain id revision number (5 != 6): invalid header height [cosmos/ibc-go/v3@v3.0.0/modules/light-clients/07-tendermint/types/client_state.go:130] With gas wanted: '0' and gas used: '120739' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "641"} }

[ancazamfir]

The proposal generated by hermes test CLI is wrong, as in the client state doesn't reflect the chain id change. When building the upgrade_height here:

hermes/crates/relayer/src/upgrade_chain.rs

Lines 71 to 74 in 097c5df

	let upgrade_height = dst_chain
	.query_latest_height() // FIXME(romac): Use query_chain_latest_height once added to ChainHandle
	.map_err(UpgradeChainError::query)?
	.add(opts.height_offset);

we must also make sure:

upgrade_height.revision_number = opts.upgraded_chain_id.version();

[ancazamfir]

just tested with temp fix in anca/3057_upgrade_chain_fix branch (5235a95) and it seems ok.