ERRO the genesis block hash for chain tevmos has changed from X to Y since the last time we ran, component: BlockStore

[vrde]

Do you want to request a feature or report a bug?

bug

What is the current behavior?

I'm running a self hosted instance connected to TEVMOS. I use docker compose I get: ERRO the genesis block hash for chain tevmos has changed from c1d935da5a718994263a95c6e811b869b2f4225638a85d34520af70176dab6c0 to 0x64ebab8b5ca43670c08744902bece6feb9e4b2fd2af998091be7eb4fc8bfd4cb since the last time we ran, component: BlockStore.

Full log

subgraph-graph-node-1  | Jun 13 13:06:51.870 INFO Graph Node version: 0.26.0 (2022-04-05)
subgraph-graph-node-1  | Jun 13 13:06:51.872 WARN GRAPH_POI_ACCESS_TOKEN not set; might leak POIs to the public via GraphQL
subgraph-graph-node-1  | Jun 13 13:06:51.874 INFO Generating configuration from command line arguments
subgraph-graph-node-1  | Jun 13 13:06:51.875 WARN No fork base URL specified, subgraph forking is disabled
subgraph-graph-node-1  | Jun 13 13:06:51.876 INFO Starting up
subgraph-graph-node-1  | Jun 13 13:06:51.877 INFO Trying IPFS node at: http://ipfs:5001/
subgraph-graph-node-1  | Jun 13 13:06:51.882 INFO Creating transport, capabilities: archive, traces, url: https://eth.bd.evmos.dev:8545, provider: tevmos-rpc-0
subgraph-graph-node-1  | Jun 13 13:06:51.887 INFO Successfully connected to IPFS node at: http://ipfs:5001/
subgraph-graph-node-1  | Jun 13 13:06:52.327 INFO Connecting to Postgres, weight: 1, conn_pool_size: 10, url: postgresql://graph-node:HIDDEN_PASSWORD@postgres:5432/graph-node, pool: main, shard: primary
subgraph-graph-node-1  | Jun 13 13:06:52.328 INFO Pool successfully connected to Postgres, pool: main, shard: primary, component: Store
subgraph-graph-node-1  | Jun 13 13:06:52.354 INFO Setting up fdw, pool: main, shard: primary, component: ConnectionPool
subgraph-graph-node-1  | Jun 13 13:06:52.360 INFO Running migrations, pool: main, shard: primary, component: ConnectionPool
subgraph-graph-node-1  | Jun 13 13:06:52.364 INFO Migrations finished, pool: main, shard: primary, component: ConnectionPool
subgraph-graph-node-1  | Jun 13 13:06:52.366 INFO Mapping primary, pool: main, shard: primary, component: ConnectionPool
subgraph-graph-node-1  | Jun 13 13:06:52.406 INFO Connecting to Ethereum to get network identifier, capabilities: archive, traces, provider: tevmos-rpc-0
subgraph-graph-node-1  | Jun 13 13:06:52.593 INFO Connected to Ethereum, capabilities: archive, traces, network_version: 9000, provider: tevmos-rpc-0
subgraph-graph-node-1  | Jun 13 13:06:52.598 ERRO the genesis block hash for chain tevmos has changed from c1d935da5a718994263a95c6e811b869b2f4225638a85d34520af70176dab6c0 to 0x64ebab8b5ca43670c08744902bece6feb9e4b2fd2af998091be7eb4fc8bfd4cb since the last time we ran, component: BlockStore
subgraph-graph-node-1  | Jun 13 13:06:52.601 INFO Creating LoadManager in disabled mode, component: LoadManager
subgraph-graph-node-1  | Jun 13 13:06:52.602 INFO Starting block ingestors with 1 chains [tevmos]
subgraph-graph-node-1  | Jun 13 13:06:52.602 ERRO Not starting block ingestor (chain is defective), network_name: tevmos
subgraph-graph-node-1  | Jun 13 13:06:52.603 INFO Starting firehose block ingestors with 0 chains []
subgraph-graph-node-1  | Jun 13 13:06:52.604 INFO Starting firehose block ingestors with 0 chains []
subgraph-graph-node-1  | Jun 13 13:06:52.604 INFO Starting job runner with 4 jobs, component: JobRunner
subgraph-graph-node-1  | Jun 13 13:06:52.611 INFO Starting JSON-RPC admin server at: http://localhost:8020, component: JsonRpcServer
subgraph-graph-node-1  | Jun 13 13:06:52.615 INFO Starting GraphQL HTTP server at: http://localhost:8000, component: GraphQLServer
subgraph-graph-node-1  | Jun 13 13:06:52.616 INFO Starting index node server at: http://localhost:8030, component: IndexNodeServer
subgraph-graph-node-1  | Jun 13 13:06:52.616 INFO Starting GraphQL WebSocket server at: ws://localhost:8001, component: SubscriptionServer
subgraph-graph-node-1  | Jun 13 13:06:52.617 INFO Starting metrics server at: http://localhost:8040, component: MetricsServer

My configuration is here https://github.com/TelediskoDAO/subgraph/tree/fix/7/evmos-compatibility

If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem.

Hopefully running my repo will give you the same result.

[vrde]

If I delete ./data (that contains ipfs and postgres data) I still have the issue. If I remove all containers then it starts from scratch again.

[jrocco2]

I'm having the same issue with a local hardhat node

[tomasz90]

Same here, any solution???

[maoueh]

When graph-node connects to a configured network, it issue a JSON-RPC request to get genesis block hash: curl -H 'Content-Type: application/json' -d '{"params":["0x0",false],"method":"eth_getBlockByNumber","jsonrpc":"2.0","id":"0x1"}' <provider_url>.

The first time it sees this network, it saves the received block's hash into the database as the network's genesis block hash. The next and upcoming times graph-node restarts, it reconnects again to a provider of the network and issue the same request. If the received block hash is different than the one saved in database, it complains.

I used https://eth.bd.evmos.dev:8545 to issue the initial request and it complains that available height is 709050

{"jsonrpc":"2.0","id":"0x1","error":{"code":-32000,"message":"height 1 is not available, lowest height is 709050"}}

So it's rather strange as the request seems to fail here. Maybe you have GRAPH_ETHEREUM_GENESIS_BLOCK_NUMBER set, in this case if this value changed over 2 restart, it will cause the shown error.

From the error message, it says that:

• Blockc1d935da5a718994263a95c6e811b869b2f4225638a85d34520af70176dab6c0:

  curl -s -H 'Content-Type: application/json' -d '{"params":["0xc1d935da5a718994263a95c6e811b869b2f4225638a85d34520af70176dab6c0",false],"method":"eth_getBlockByHash","jsonrpc":"2.0","id":"0x1"}' https://eth.bd.evmos.dev:8545 | jq -r '"\(.result.number)"' | to_dec
  1 671 525
  

• Changed to 0x64ebab8b5ca43670c08744902bece6feb9e4b2fd2af998091be7eb4fc8bfd4cb:

  curl -s -H 'Content-Type: application/json' -d '{"params":["0x64ebab8b5ca43670c08744902bece6feb9e4b2fd2af998091be7eb4fc8bfd4cb",false],"method":"eth_getBlockByHash","jsonrpc":"2.0","id":"0x1"}' https://eth.bd.evmos.dev:8545 | jq -r '"\(.result.number)"' | to_dec
  1 715 751
  

So either by misconfiguration of GRAPH_ETHEREUM_GENESIS_BLOCK_NUMBER or by a a wrong inconsistent JSON-RPC response from one (or different) providers, the response of curl -H 'Content-Type: application/json' -d '{"params":["0x0",false],"method":"eth_getBlockByNumber","jsonrpc":"2.0","id":"0x1"}' <provider_url> over 2 or more starts of graph-node led to different block hash being returned.

I'll close as not a bug until further evidence shows the contrary. Don't hesitate to continue the conversation for further investigation.

[songhobby]

I think the initial block hash is stored in the container settings the first time the containers run. Deleting the containers solve the problem but deleting the mapped volume does not.

Solution:

docker-compose down
docker container prune

[BlockCrafterX]

Solution :
docker-compose rm -f
docker-compose pull
docker-compose up --build -d

[paymog]

I'm seeing this issue happen in production for fantom, what's the right way to fix it?

Jan 05 13:00:00.269 ERRO the genesis block hash for chain fantom has changed from 00000000000003e83fddf1e9330f0a8691d9f0b2af57b38c3bb85488488a40df to 0x0000000000000000c20dbfb2ec18ae20037c716f3ba2d9e1da768a9deca17cb4 since the last time we ran, component: BlockStore