Getting client state was not found error when querying client status on Celestia testnet

[rootulp]

Context

I'm trying to set up IBC clients between two tesnets:

1. https://github.com/cosmos/chain-registry/blob/master/testnets/cosmoshubtestnet/chain.json
2. https://github.com/cosmos/chain-registry/blob/master/testnets/celestiatestnet3/chain.json

Problem

I created a client with

hermes create client --host-chain mocha-4 --reference-chain theta-testnet-001

now when I query it's state, I observe:

$ hermes query client status --chain mocha-4 --client theta-testnet-001
2023-09-16T17:34:42.944610Z  INFO ThreadId(01) using default configuration from '/Users/rootulp/.hermes/config.toml'
2023-09-16T17:34:42.945007Z  INFO ThreadId(01) running Hermes v1.6.0+1c1cf02
2023-09-16T17:34:43.618969Z  WARN ThreadId(01) Unsupported tendermint version, will use v0.37 compatibility mode but relaying might not work as desired: unsupported Tendermint version reported by the node: 1.0.0-rc14
ERROR error decoding protobuf: error converting message type into domain type: the client state was not found

Given that error message, I'm not sure how to resolve.

[rootulp]

Note this still repros if I fallback to v0.34 compat mode

$ ./target/release/hermes query client status --chain mocha-4 --client theta-testnet-001
2023-09-16T19:38:09.229157Z  INFO ThreadId(01) using default configuration from '/Users/rootulp/.hermes/config.toml'
2023-09-16T19:38:09.231700Z  INFO ThreadId(01) running Hermes v1.6.0+1c1cf0298-dirty
2023-09-16T19:38:09.914472Z  WARN ThreadId(01) Unsupported tendermint version, will use v0.34 compatibility mode but relaying might not work as desired: unsupported Tendermint version reported by the node: 1.0.0-rc14
ERROR error decoding protobuf: error converting message type into domain type: the client state was not found

[romac]

This error can occur if the client state returned by the node, encoded as a Protobuf Any type, does not include a type URL and thus cannot be decoded. We report that the client state was not found because when that happens, the encoded byte array is also usually empty.

I am not sure what's going wrong exactly with this client yet, we need to investigate further.

In the meantime, if you could send us the full logs you get when you create the client that would be much appreciated!

[rootulp]

Thanks for addressing all my feedback! I just tried to repro this error but this time I created a new client using my patch of hermes that falls back to v0.34 compat mode. When I do that, things work so I think my original issue may have been that I was trying to query a client that was created without my compat mode patch (?). Happy to close this issue as I think the root cause is already being fixed via #3623

Create client

$ ./target/release/hermes create client --host-chain mocha-4 --reference-chain theta-testnet-001
2023-10-05T14:12:53.023867Z  INFO ThreadId(01) using default configuration from '/Users/rootulp/.hermes/config.toml'
2023-10-05T14:12:53.025645Z  INFO ThreadId(01) running Hermes v1.6.0+1c1cf0298-dirty
2023-10-05T14:12:54.841219Z  WARN ThreadId(01) Unsupported tendermint version, will use v0.34 compatibility mode but relaying might not work as desired: unsupported Tendermint version reported by the node: 1.0.0-rc16
SUCCESS CreateClient(
    CreateClient(
        Attributes {
            client_id: ClientId(
                "07-tendermint-4",
            ),
            client_type: Tendermint,
            consensus_height: Height {
                revision: 0,
                height: 18233959,
            },
        },
    ),
)

Query client

$ ./target/release/hermes query client status --chain mocha-4 --client 07-tendermint-4
2023-10-05T14:13:53.099260Z  INFO ThreadId(01) using default configuration from '/Users/rootulp/.hermes/config.toml'
2023-10-05T14:13:53.099473Z  INFO ThreadId(01) running Hermes v1.6.0+1c1cf0298-dirty
2023-10-05T14:13:53.755571Z  WARN ThreadId(01) Unsupported tendermint version, will use v0.34 compatibility mode but relaying might not work as desired: unsupported Tendermint version reported by the node: 1.0.0-rc16
SUCCESS Active

[romac]

Glad to hear, thanks for letting us know!