Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Various errors with Neutron Network #17

Closed
qwertzlbert opened this issue Dec 16, 2024 · 5 comments
Closed

[BUG] Various errors with Neutron Network #17

qwertzlbert opened this issue Dec 16, 2024 · 5 comments
Assignees
Labels
bug Something isn't working

Comments

@qwertzlbert
Copy link
Contributor

Problem Description

Using CVMS to monitor Validators on the Neutron network won't work due to various issues.

  1. uptime and missed blocks collection will fail
  2. Lookup of consumer chains fail

Steps to Reproduce

add a config to monitor a neutron validator. E.g.

monikers:
  - 'Figment'
  chains:
  - display_name: 'neutron'
    chain_id: 'neutron-1'
    tracking_addresses:
      - 'neutron1hjct6q7npsspsg3dgvzk3sdf89spmlpfvc8vne'
    nodes:
      - rpc: 'https://neutron-rpc.polkachu.com'
        api: 'https://neutron-api.polkachu.com'
        grpc: 'neutron-grpc.polkachu.com:19190'

Start CVMS
This will result in the following errors:

ERROR   api error: failed to get specific denom balnce  chain="neutron" chain_id="neutron-1" package="balance"
ERROR   failed to collect all accounts balances, got errors count: 1  chain="neutron" chain_id="neutron-1" package="balance"
ERROR   failed to update metrics: cvms common errors: failed to request from node 
...
ERROR   failed to update metrics err: failed to get consumer chain id: context deadline exceeded and going to sleep 10s...  chain="neutron" chain_id="neutron-1" package="uptime"
...
ERROR   failed to update metrics err: failed to get consumer chain id: context deadline exceeded and going to sleep 10s...  chain="neutron" chain_id="neutron-1" package="uptime"

Patching the issue related to the de denom lookup will return the following error

ERROR   failed to update metrics err: got 501 code from https://neutron-api.polkachu.com/cosmos/staking/v1beta1/validators?status=BOND_STATUS_BONDED&pagination.count_total=true&pagination.limit=500 and going to sleep 10s...  chain="neutron" chain_id="neutron-1" package="uptime"

Expected Behavior

Missed blocks for the validator and consumer chain metrics are exported via prometheus.

Additional Information

Issue 1 is related to the staking API not being supported by Neutron, this means the valcons address can't be looked up via /cosmos/staking/v1beta1/validators?status=BOND_STATUS_BONDED&pagination.count_total=true&pagination.limit=500
I tried to fix this, but seems like there's no easy way to actually lookup the valcons address on neutron other than using the tendermint CLI query on the validator itself (tendermint show-address) which is obviously no solution. Alternatively it could either be required for Neutron to use the valcons key instead of the normal account address.

Issue 2 is caused by Neutron not implementing the ccv/provider API query which is used by CVMS to lookup the consumer (

var ConsumerChainListQueryPath string = "/interchain_security/ccv/provider/consumer_chains/3"
) chains. Instead it only implements ccv/consumer calls.

Until a fix is available I would recommend to either remove support for Neutron entirely or just remove support for the uptime module

@qwertzlbert qwertzlbert added the bug Something isn't working label Dec 16, 2024
@carameleon
Copy link
Member

Hey @qwertzlbert ! Thank you for reporting this. We will look into it. Happy X-mas!

@qwertzlbert
Copy link
Contributor Author

qwertzlbert commented Dec 29, 2024

Thank you very much! Same back to you!


I was also taking a look at stride just before Christmas and unfortunately the issue is relatively similar here. The network misses the required API methods via RPC but seems to only support the required requests via gRPC.
I took a look at the implementation of the uptime module and started to refactor it a bit to be a bit more flexible if needed, basically following the style of the other modules where this flexibility is required.
I'll link it here when I made some progress

@qwertzlbert
Copy link
Contributor Author

I had some time this week to take a look at consumer chains (mostly stride and neutron) and figured this might get a bit more complicated than I thought.
As consumer chains use validator addresses on different cosmos chains as security, meaning those will get penalized instead of the validator address on the consumer chain, the lookup and tracking needs to be done across different chains.

Doing it like it's implemented right now won't work, as first the consumer chain does not implement /interchain_security/ccv/provider/consumer_chains/ (https://buf.build/cosmos/interchain-security/docs/main:interchain_security.ccv.provider.v1) but only partially https://buf.build/cosmos/interchain-security/docs/main:interchain_security.ccv.consumer.v1 and second the returned staking validators from /cosmos/staking/v1beta1/validators (https://buf.build/cosmos/cosmos-sdk/docs/main:cosmos.staking.v1beta1) is not the correct list of validators
E.g. In the case of stride the consumer feature was activated after chain launch so it had some staking validators initially the request to https://buf.build/cosmos/cosmos-sdk/docs/main:cosmos.staking.v1beta1 will return an outdated list of old validators and pubkeys not necessarily used as signer addresses anymore (the validators you get when querying the tendermint RPC via /validators)

To actually resolve the monikers and relationship between signer address and security address you have to look up the linked consumer chain on the provider chain. In the case of stride and neutron this is cosmoshub-4.

On cosmos hub you will find the interchain_security.ccv.provider.v1 interface (https://buf.build/cosmos/interchain-security/docs/main:interchain_security.ccv.provider.v1) you can use to lookup the consumerchains an individual cosmoshub-4 validator secures.
E.g. if we want to look up all the validators on cosmoshub-4 securing stride we have to do a query to the QueryConsumerValidators endpoint (https://buf.build/cosmos/interchain-security/docs/main:interchain_security.ccv.provider.v1#interchain_security.ccv.provider.v1.QueryConsumerValidatorsRequest) with {"chain_id": "1"}as payload. This will return the following list of validators

{
    "validators": [
    .
    .
    .
        {
            "provider_address": "cosmosvalcons1m8u2gxmc92n2v6kus8u48y3u0h88kcqp69cr3a",
            "consumer_key": {
                "ed25519": "dDnmUif7kgHxWulmkQPT87PcUs8MoROEOq44FRhkokM="
            },
            "power": "0",
            "rate": "0",
            "consumer_power": "1943256",
            "consumer_commission_rate": "90000000000000000",
            "provider_commission_rate": "90000000000000000",
            "description": {
                "moniker": "Figment",
                "identity": "E5F274B870BDA01D",
                "website": "https://figment.io",
                "security_contact": "",
                "details": "Makers of Hubble and Canada’s largest Cosmos validator, Figment is the easiest and most secure way to stake your Atoms."
            },
            "provider_operator_address": "cosmosvaloper1hjct6q7npsspsg3dgvzk3sdf89spmlpfdn6m9d",
            "jailed": false,
            "status": "BOND_STATUS_BONDED",
            "provider_tokens": "1943256935480",
            "provider_power": "1943256",
            "validates_current_epoch": true
        },
...

Looking up the provider_operators_address will give you the Figment validator on cosmoshub-4 (https://www.mintscan.io/cosmos/validators/cosmosvaloper1hjct6q7npsspsg3dgvzk3sdf89spmlpfdn6m9d/)

Now we can take the consumer_key (pubkey) from this output and encode it into a bech32 consensus address for stride which uses the HRP stridevalcons, this will return stridevalcons1wgalx4ahmcdelgjsssl360sk3dsw702qeg8sdp for the validator above.
If you now compare this address and pubkey with the list of last singers returned by https://stride-rpc.publicnode.com:443/validators you will see the pubkey from cosmoshub-4 (dDnmUif7kgHxWulmkQPT87PcUs8MoROEOq44FRhkokM=) matches the following entry:

.
.
.
            {
                "address": "723BF357B7DE1B9FA250843F1D3E168B60EF3D40",
                "pub_key": {
                    "type": "tendermint/PubKeyEd25519",
                    "value": "dDnmUif7kgHxWulmkQPT87PcUs8MoROEOq44FRhkokM="
                },
                "voting_power": "1943256",
                "proposer_priority": "45370160"
            }
            
.
.
.

And now we can finally look up the signer stats we want to export via the signer_infos api e.g. via https://stride-rest.publicnode.com:443/cosmos/slashing/v1beta1/signing_infos/stridevalcons1wgalx4ahmcdelgjsssl360sk3dsw702qeg8sdp which returns the validator and its current stats we can use for the uptime metric.

{
    "val_signing_info": {
        "address": "stridevalcons1wgalx4ahmcdelgjsssl360sk3dsw702qeg8sdp",
        "start_height": "4616678",
        "index_offset": "10046366",
        "jailed_until": "1970-01-01T00:00:00Z",
        "tombstoned": false,
        "missed_blocks_counter": "754"
    }
}

So to summarize what we need to do to get the stats we need here:

  1. lookup the supported consumer chains on the provider chain (e.g. cosmoshub-4)
  2. get the validators backing a signer on the consumer chain
  3. get the used pubkey
  4. query the current list of signers from the consumer chain (e.g. stride)
  5. match both pubkeys or encoded valcons addresses
  6. get the signing infos for this address from the consumer chain

Due to this I'm not quite sure how we could implement the support for such networks. I know you started the work already, but I was not able to get it working for now. I'm still playing around trying to find a working solution which makes sense, but again as I said above not as straight forward as I thought.

Also it probably makes sense to document this somewhere :)

@qwertzlbert
Copy link
Contributor Author

qwertzlbert commented Jan 8, 2025

Alright I think this was a layer 8 problem on my end and the issue was with the config I created
Using the correct config like the one below will actually do the right thing.

monikers:
  - 'Figment'
  - 'figment'
  - 'Cosmostation'
chains:
  - display_name: 'stride'
    chain_id: 'stride-1'
    tracking_addresses:
      - 'stride1hjct6q7npsspsg3dgvzk3sdf89spmlpftvwjaj'
      - 'stride1clpqr4nrk4khgkxj78fcwwh6dl3uw4epkqgq3v'
    nodes:
      - rpc: 'https://stride-rpc.publicnode.com:443'
        api: 'https://stride-rest.publicnode.com'
        grpc: 'stride-grpc.polkachu.com:12290'

      - rpc: 'https://stride-rpc.polkachu.com'
        api: 'https://stride-api.polkachu.com'
        grpc: 'stride-grpc.polkachu.com:12290'

      - rpc: 'https://rpc.lavenderfive.com:443/stride'
        api: 'https://rest.lavenderfive.com:443/stride'
        grpc: 'stride.lavenderfive.com:443'
    provider_nodes:
      - rpc: 'https://cosmos-rpc.publicnode.com:443'
        api: 'https://cosmos-rest.publicnode.com'
        grpc: 'cosmos-grpc.polkachu.com:14990'

      - rpc: 'https://cosmos-rpc.polkachu.com:443'
        grpc: 'cosmos-grpc.polkachu.com:14990'
        api: 'https://cosmos-api.polkachu.com'

I did not realize you had already implemented this correctly as I did not understand the provider_nodes config option correctly. I think I will add some docs related to this and the endpoints used

@qwertzlbert
Copy link
Contributor Author

qwertzlbert commented Jan 9, 2025

I created a PR including a readme explaining the consumer chain support in a bit more detail here: #23
Given the issue I opened is invalid you can probably close this issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants