Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] Panic on startup #4655

Closed
2 of 3 tasks
paymog opened this issue May 28, 2023 · 4 comments · Fixed by #4680
Closed
2 of 3 tasks

[Bug] Panic on startup #4655

paymog opened this issue May 28, 2023 · 4 comments · Fixed by #4680
Labels
bug Something isn't working

Comments

@paymog
Copy link

paymog commented May 28, 2023

Bug report

I'm running graph node in a docker compose set and I've noticed that v0.31.0-rc.1 reliably panics on startup. Here are all the logs I see when starting with this image.

Here's my toml file:

[store]
[store.primary]
connection = "postgresql://graph-node:let-me-in@postgres:5432/client__dev"
pool_size = 10
[store.second]
connection = "postgresql://graph-node:let-me-in@postgres-second:5432/client__dev"
pool_size = 10

[deployment]
[[deployment.rule]]
shard = ["primary", "second"]
indexers = [ "default" ]

[chains]
ingestor = "default"
[chains.mainnet]
shard = "primary"
provider = [
    { label = "mainnet-default", transport = "rpc", url = "http://host.docker.internal:3000/mainnet/default", features = [] },
    { label = "mainnet-archive", transport = "rpc", url = "http://host.docker.internal:3000/mainnet/archive", features = ["archive"] },
    { label = "mainnet-archive-traces", transport = "rpc", url = "http://host.docker.internal:3000/mainnet/archiveAndTraces", features = ["traces", "archive"] },
    { label = "streamingfast-firehose-mainnet", details = { type = "firehose", url = "https://mainnet.eth.streamingfast.io", features = [ "filters" ], token = "<token>" }},
]
[chains.avalanche]
shard = "primary"
provider = [
    { label = "avalanche-default", transport = "rpc", url = "http://host.docker.internal:3000/avalanche/default", features = [] },
]

Here are the env vars I use when starting up:

    environment:
      GRAPH_NODE_CONFIG: /etc/graphnode.toml
      ipfs: "ipfs:5001"
      GRAPH_LOG: debug
      node_role: "combined-node"

Relevant log output

e2e-tests-graph-node-1             | May 28 13:35:04.716 INFO Graph Node version: 0.31.0 (2023-05-26)
e2e-tests-graph-node-1             | May 28 13:35:04.718 WARN GRAPH_POI_ACCESS_TOKEN not set; might leak POIs to the public via GraphQL
e2e-tests-graph-node-1             | May 28 13:35:04.720 INFO Reading configuration file `/etc/graphnode.toml`
e2e-tests-graph-node-1             | May 28 13:35:04.721 WARN No fork base URL specified, subgraph forking is disabled
e2e-tests-graph-node-1             | May 28 13:35:04.721 INFO Starting up
e2e-tests-graph-node-1             | May 28 13:35:04.721 INFO Trying IPFS node at: http://ipfs:5001/
e2e-tests-graph-node-1             | May 28 13:35:04.741 INFO Creating transport, capabilities: , url: http://host.docker.internal:3000/avalanche/default, provider: avalanche-default
e2e-tests-graph-node-1             | May 28 13:35:04.746 INFO Successfully connected to IPFS node at: http://ipfs:5001/
e2e-tests-graph-node-1             | May 28 13:35:04.750 INFO Creating transport, capabilities: , url: http://host.docker.internal:3000/mainnet/default, provider: mainnet-default
e2e-tests-graph-node-1             | May 28 13:35:04.989 INFO Creating transport, capabilities: archive, url: http://host.docker.internal:3000/mainnet/archive, provider: mainnet-archive
e2e-tests-graph-node-1             | May 28 13:35:05.299 INFO Creating transport, capabilities: archive, traces, url: http://host.docker.internal:3000/mainnet/archiveAndTraces, provider: mainnet-archive-traces
e2e-tests-graph-node-1             | May 28 13:35:06.059 DEBG Creating firehose networks [2 chains, ingestor default]
e2e-tests-graph-node-1             | May 28 13:35:06.059 INFO Configuring firehose endpoint, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:06.068 DEBG Creating firehose networks [2 chains, ingestor default]
e2e-tests-graph-node-1             | May 28 13:35:06.069 WARN Expensive queries file not set to a valid file: /etc/graph-node/expensive-queries.txt
e2e-tests-graph-node-1             | May 28 13:35:06.069 DEBG Cleaning up large notifications after about 300s, channel: store_events, component: NotificationListener
e2e-tests-graph-node-1             | May 28 13:35:06.087 INFO Connecting to Postgres, weight: 1, conn_pool_size: 10, url: postgresql://graph-node:HIDDEN_PASSWORD@postgres:5432/client__dev, pool: main, shard: primary
e2e-tests-graph-node-1             | May 28 13:35:06.089 INFO Pool successfully connected to Postgres, pool: main, shard: primary, component: Store
e2e-tests-graph-node-1             | May 28 13:35:06.091 INFO Connecting to Postgres, weight: 1, conn_pool_size: 10, url: postgresql://graph-node:HIDDEN_PASSWORD@postgres-second:5432/client__dev, pool: main, shard: second
e2e-tests-graph-node-1             | May 28 13:35:06.094 INFO Pool successfully connected to Postgres, pool: main, shard: second, component: Store
e2e-tests-graph-node-1             | May 28 13:35:06.098 DEBG Using postgres host order [Main], shard: primary, component: Store
e2e-tests-graph-node-1             | May 28 13:35:06.098 DEBG Using postgres host order [Main], shard: second, component: Store
e2e-tests-graph-node-1             | May 28 13:35:06.106 INFO Setting up fdw, pool: main, shard: primary, component: ConnectionPool
e2e-tests-graph-node-1             | May 28 13:35:06.115 INFO Setting up fdw, pool: main, shard: second, component: ConnectionPool
e2e-tests-graph-node-1             | May 28 13:35:06.130 INFO Running migrations, pool: main, shard: second, component: ConnectionPool
e2e-tests-graph-node-1             | May 28 13:35:06.143 INFO Running migrations, pool: main, shard: primary, component: ConnectionPool
e2e-tests-graph-node-1             | May 28 13:35:07.004 INFO Migrations finished, pool: main, shard: second, component: ConnectionPool
e2e-tests-graph-node-1             | May 28 13:35:07.006 DEBG Postgres migration output, output: Running migration 20180710061642 Running migration 20180710061659 Running migration 20180710062717 Running migration 20180710062730 Running migration 20180816143800 Running migration 20180822130000 Running migration 20180822140000 Running migration 20180829120000 Running migration 20180907220000 Running migration 20180918180000 Running migration 20181202114100 Running migration 20181203150000 Running migration 20181221003727 Running migration 20190107120000 Running migration 20190118195346 Running migration 20190118195526 Running migration 20190201045927 Running migration 20190204114900 Running migration 20190225182843 Running migration 20190226182914 Running migration 20190226183156 Running migration 20190227035443 Running migration 20190228190800 Running migration 20190304235349 Running migration 20190307171355 Running migration 20190328004319 Running migration 20190331010824 Running migration 20190419171709 Running migration 20190422190022 Running migration 20190503164052 Running migration 20190509135900 Running migration 20190509232642 Running migration 20190514200255 Running migration 20190515215022 Running migration 20190605214320 Running migration 20190626164405 Running migration 20190720195916 Running migration 20190802001120 Running migration 20190901135850 Running migration 20190902230613 Running migration 20191001173616 Running migration 20200114235608 Running migration 20200117013633 Running migration 20200124065338 Running migration 20200306020253 Running migration 20200311162100 Running migration 20200325170527 Running migration 20200404002817 Running migration 20200410111111 Running migration 20200516225611 Running migration 20200707002933 Running migration 20200731162138 Running migration 20201031150000 Running migration 20201103170839 Running migration 20201110100000 Running migration 20201211142000 Running migration 20201212000001 Running migration 20201212000002 Running migration 20201212000003 Running migration 20201212000004 Running migration 20201212000005 Running migration 20201215000000 Running migration 2020127190800 Running migration 20210107004939 Running migration 20210114175654 Running migration 20210114193022 Running migration 20210115013524 Running migration 20210119033749 Running migration 20210126044443 Running migration 20210126172953 Running migration 20210126173710 Running migration 20210126204036 Running migration 20210217205502 Running migration 20210218000721 Running migration 20210218171423 Running migration 20210224051050 Running migration 20210225233156 Running migration 20210311010830 Running migration 20210311231340 Running migration 20210312014815 Running migration 20210312070453 Running migration 20210316001809 Running migration 20210316165131 Running migration 20210319161012 Running migration 20210320001347 Running migration 20210324220541 Running migration 20210520130220 Running migration 20210521165219 Running migration 20210601222649 Running migration 20210614201635 Running migration 20210709144144 Running migration 20211202212506 Running migration 20211205061157 Running migration 2022030907000 Running migration 20220321172028 Running migration 20220323010300 Running migration 20220426125552 Running migration 20220612003442 Running migration 20220919161239 Running migration 20221013000000 Running migration 20221103213140 Running migration 20221110185105 Running migration 20230124192319 Running migration 20230215184255 Running migration 20230306002954 Running migration 20230306233030 Running migration 20230328004152, pool: main, shard: second, component: ConnectionPool
e2e-tests-graph-node-1             | May 28 13:35:07.006 INFO Mapping metadata from second, pool: main, shard: primary, component: ConnectionPool
e2e-tests-graph-node-1             | May 28 13:35:07.059 INFO Migrations finished, pool: main, shard: primary, component: ConnectionPool
e2e-tests-graph-node-1             | May 28 13:35:07.063 DEBG Postgres migration output, output: Running migration 20180710061642 Running migration 20180710061659 Running migration 20180710062717 Running migration 20180710062730 Running migration 20180816143800 Running migration 20180822130000 Running migration 20180822140000 Running migration 20180829120000 Running migration 20180907220000 Running migration 20180918180000 Running migration 20181202114100 Running migration 20181203150000 Running migration 20181221003727 Running migration 20190107120000 Running migration 20190118195346 Running migration 20190118195526 Running migration 20190201045927 Running migration 20190204114900 Running migration 20190225182843 Running migration 20190226182914 Running migration 20190226183156 Running migration 20190227035443 Running migration 20190228190800 Running migration 20190304235349 Running migration 20190307171355 Running migration 20190328004319 Running migration 20190331010824 Running migration 20190419171709 Running migration 20190422190022 Running migration 20190503164052 Running migration 20190509135900 Running migration 20190509232642 Running migration 20190514200255 Running migration 20190515215022 Running migration 20190605214320 Running migration 20190626164405 Running migration 20190720195916 Running migration 20190802001120 Running migration 20190901135850 Running migration 20190902230613 Running migration 20191001173616 Running migration 20200114235608 Running migration 20200117013633 Running migration 20200124065338 Running migration 20200306020253 Running migration 20200311162100 Running migration 20200325170527 Running migration 20200404002817 Running migration 20200410111111 Running migration 20200516225611 Running migration 20200707002933 Running migration 20200731162138 Running migration 20201031150000 Running migration 20201103170839 Running migration 20201110100000 Running migration 20201211142000 Running migration 20201212000001 Running migration 20201212000002 Running migration 20201212000003 Running migration 20201212000004 Running migration 20201212000005 Running migration 20201215000000 Running migration 2020127190800 Running migration 20210107004939 Running migration 20210114175654 Running migration 20210114193022 Running migration 20210115013524 Running migration 20210119033749 Running migration 20210126044443 Running migration 20210126172953 Running migration 20210126173710 Running migration 20210126204036 Running migration 20210217205502 Running migration 20210218000721 Running migration 20210218171423 Running migration 20210224051050 Running migration 20210225233156 Running migration 20210311010830 Running migration 20210311231340 Running migration 20210312014815 Running migration 20210312070453 Running migration 20210316001809 Running migration 20210316165131 Running migration 20210319161012 Running migration 20210320001347 Running migration 20210324220541 Running migration 20210520130220 Running migration 20210521165219 Running migration 20210601222649 Running migration 20210614201635 Running migration 20210709144144 Running migration 20211202212506 Running migration 20211205061157 Running migration 2022030907000 Running migration 20220321172028 Running migration 20220323010300 Running migration 20220426125552 Running migration 20220612003442 Running migration 20220919161239 Running migration 20221013000000 Running migration 20221103213140 Running migration 20221110185105 Running migration 20230124192319 Running migration 20230215184255 Running migration 20230306002954 Running migration 20230306233030 Running migration 20230328004152, pool: main, shard: primary, component: ConnectionPool
e2e-tests-graph-node-1             | May 28 13:35:07.121 DEBG Release migration lock, pool: main, shard: second, component: ConnectionPool
e2e-tests-graph-node-1             | May 28 13:35:07.121 INFO Mapping primary, pool: main, shard: primary, component: ConnectionPool
e2e-tests-graph-node-1             | May 28 13:35:07.123 DEBG Setup finished, setup_time_s: 1, pool: main, shard: second, component: ConnectionPool
e2e-tests-graph-node-1             | May 28 13:35:07.125 INFO Mapping primary, pool: main, shard: second, component: ConnectionPool
e2e-tests-graph-node-1             | May 28 13:35:07.162 INFO Mapping metadata from primary, pool: main, shard: second, component: ConnectionPool
e2e-tests-graph-node-1             | May 28 13:35:07.239 DEBG Release migration lock, pool: main, shard: primary, component: ConnectionPool
e2e-tests-graph-node-1             | May 28 13:35:07.241 DEBG Setup finished, setup_time_s: 1, pool: main, shard: primary, component: ConnectionPool
e2e-tests-graph-node-1             | May 28 13:35:07.241 DEBG Cleaning up large notifications after about 300s, channel: chain_head_updates, component: ChainHeadUpdateListener > NotificationListener
e2e-tests-graph-node-1             | May 28 13:35:07.248 INFO Connecting to Ethereum to get network identifier, capabilities: , provider: avalanche-default
e2e-tests-graph-node-1             | May 28 13:35:07.249 INFO Connecting to Ethereum to get network identifier, capabilities: , provider: mainnet-default
e2e-tests-graph-node-1             | May 28 13:35:07.249 INFO Connecting to Ethereum to get network identifier, capabilities: archive, provider: mainnet-archive
e2e-tests-graph-node-1             | May 28 13:35:07.253 INFO Connecting to Ethereum to get network identifier, capabilities: archive, traces, provider: mainnet-archive-traces
e2e-tests-graph-node-1             | May 28 13:35:07.503 INFO Connected to Ethereum, capabilities: archive, traces, network_version: 1, provider: mainnet-archive-traces
e2e-tests-graph-node-1             | May 28 13:35:07.514 INFO Connected to Ethereum, capabilities: , network_version: 43114, provider: avalanche-default
e2e-tests-graph-node-1             | May 28 13:35:07.515 INFO Connected to Ethereum, capabilities: , network_version: 1, provider: mainnet-default
e2e-tests-graph-node-1             | May 28 13:35:07.934 INFO Connected to Ethereum, capabilities: archive, network_version: 1, provider: mainnet-archive
e2e-tests-graph-node-1             | May 28 13:35:07.934 INFO Connecting to Firehose to get chain identifier, provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.934 INFO Requesting genesis block from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.934 DEBG Connecting to firehose to retrieve block for number 0, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.934 INFO Connecting to Firehose to get chain identifier, provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.934 INFO Requesting genesis block from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.934 DEBG Connecting to firehose to retrieve block for number 0, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.934 INFO Connecting to Firehose to get chain identifier, provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.934 INFO Requesting genesis block from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.934 DEBG Connecting to firehose to retrieve block for number 0, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.934 INFO Connecting to Firehose to get chain identifier, provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.934 INFO Requesting genesis block from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.934 DEBG Connecting to firehose to retrieve block for number 0, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.934 INFO Connecting to Firehose to get chain identifier, provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.934 INFO Requesting genesis block from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.934 DEBG Connecting to firehose to retrieve block for number 0, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.934 INFO Connecting to Firehose to get chain identifier, provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.934 INFO Requesting genesis block from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.935 DEBG Connecting to firehose to retrieve block for number 0, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.935 INFO Connecting to Firehose to get chain identifier, provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.935 INFO Requesting genesis block from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.935 DEBG Connecting to firehose to retrieve block for number 0, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.935 INFO Connecting to Firehose to get chain identifier, provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.935 INFO Requesting genesis block from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.935 DEBG Connecting to firehose to retrieve block for number 0, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.935 INFO Connecting to Firehose to get chain identifier, provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.935 INFO Requesting genesis block from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.935 DEBG Connecting to firehose to retrieve block for number 0, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.935 INFO Connecting to Firehose to get chain identifier, provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.935 INFO Requesting genesis block from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.935 DEBG Connecting to firehose to retrieve block for number 0, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.935 INFO Connecting to Firehose to get chain identifier, provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.935 INFO Requesting genesis block from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.935 DEBG Connecting to firehose to retrieve block for number 0, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.935 INFO Connecting to Firehose to get chain identifier, provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.935 INFO Requesting genesis block from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.935 DEBG Connecting to firehose to retrieve block for number 0, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.935 INFO Connecting to Firehose to get chain identifier, provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.935 INFO Requesting genesis block from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.935 DEBG Connecting to firehose to retrieve block for number 0, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.935 INFO Connecting to Firehose to get chain identifier, provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.935 INFO Requesting genesis block from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.935 DEBG Connecting to firehose to retrieve block for number 0, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.935 INFO Connecting to Firehose to get chain identifier, provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.935 INFO Requesting genesis block from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.935 DEBG Connecting to firehose to retrieve block for number 0, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.935 INFO Connecting to Firehose to get chain identifier, provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.935 INFO Requesting genesis block from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.935 DEBG Connecting to firehose to retrieve block for number 0, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.935 INFO Connecting to Firehose to get chain identifier, provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.935 INFO Requesting genesis block from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.935 DEBG Connecting to firehose to retrieve block for number 0, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.935 INFO Connecting to Firehose to get chain identifier, provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.935 INFO Requesting genesis block from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.935 DEBG Connecting to firehose to retrieve block for number 0, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.935 INFO Connecting to Firehose to get chain identifier, provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.935 INFO Requesting genesis block from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.935 DEBG Connecting to firehose to retrieve block for number 0, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.935 INFO Connecting to Firehose to get chain identifier, provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.935 INFO Requesting genesis block from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:07.935 DEBG Connecting to firehose to retrieve block for number 0, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:08.311 DEBG Retrieving block(s) from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:08.313 DEBG Retrieving block(s) from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:08.317 DEBG Retrieving block(s) from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:08.320 DEBG Retrieving block(s) from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:08.321 DEBG Retrieving block(s) from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:08.321 DEBG Retrieving block(s) from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:08.333 DEBG Retrieving block(s) from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:08.341 DEBG Retrieving block(s) from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:08.354 DEBG Retrieving block(s) from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:08.363 DEBG Retrieving block(s) from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:08.367 DEBG Retrieving block(s) from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:08.369 DEBG Retrieving block(s) from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:08.371 DEBG Retrieving block(s) from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:08.374 DEBG Retrieving block(s) from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:08.382 DEBG Retrieving block(s) from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:08.388 DEBG Retrieving block(s) from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:08.396 DEBG Retrieving block(s) from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:08.405 DEBG Retrieving block(s) from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:08.502 DEBG Retrieving block(s) from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:08.521 DEBG Retrieving block(s) from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:09.403 INFO Connected to Firehose, genesis_block: #0 (d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3), provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:09.445 INFO Connected to Firehose, genesis_block: #0 (d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3), provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:09.523 INFO Connected to Firehose, genesis_block: #0 (d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3), provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:09.561 INFO Connected to Firehose, genesis_block: #0 (d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3), provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:09.563 INFO Connected to Firehose, genesis_block: #0 (d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3), provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:09.656 INFO Connected to Firehose, genesis_block: #0 (d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3), provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:09.807 INFO Connected to Firehose, genesis_block: #0 (d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3), provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:09.843 INFO Connected to Firehose, genesis_block: #0 (d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3), provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:10.055 INFO Connected to Firehose, genesis_block: #0 (d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3), provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:10.075 INFO Connected to Firehose, genesis_block: #0 (d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3), provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:10.082 INFO Connected to Firehose, genesis_block: #0 (d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3), provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:10.150 INFO Connected to Firehose, genesis_block: #0 (d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3), provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:10.342 INFO Connected to Firehose, genesis_block: #0 (d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3), provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:10.348 INFO Connected to Firehose, genesis_block: #0 (d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3), provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:10.353 INFO Connected to Firehose, genesis_block: #0 (d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3), provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:10.409 INFO Connected to Firehose, genesis_block: #0 (d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3), provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:10.499 INFO Connected to Firehose, genesis_block: #0 (d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3), provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:10.755 INFO Connected to Firehose, genesis_block: #0 (d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3), provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:10.892 INFO Connected to Firehose, genesis_block: #0 (d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3), provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:11.114 INFO Connected to Firehose, genesis_block: #0 (d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3), provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | thread 'tokio-runtime-worker' panicked at 'Creating the BlockStore works: Unknown(duplicate key value violates unique constraint "chains_name_key")', node/src/store_builder.rs:191:14
e2e-tests-graph-node-1             | note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
e2e-tests-graph-node-1             | Panic in tokio task, aborting!
e2e-tests-graph-node-1             | Aborted
e2e-tests-graph-node-1 exited with code 134
e2e-tests-graph-node-1             | wait-for-it.sh: waiting 15 seconds for postgres-second:5432
e2e-tests-graph-node-1             | wait-for-it.sh: postgres-second:5432 is available after 0 seconds
e2e-tests-graph-node-1             | wait-for-it.sh: waiting 15 seconds for ipfs:5001
e2e-tests-graph-node-1             | wait-for-it.sh: ipfs:5001 is available after 0 seconds
e2e-tests-graph-node-1             | wait-for-it.sh: waiting 15 seconds for rpc-node-proxy:3000
e2e-tests-graph-node-1             | wait-for-it.sh: rpc-node-proxy:3000 is available after 0 seconds
e2e-tests-graph-node-1             | May 28 13:35:17.071 INFO Graph Node version: 0.31.0 (2023-05-26)
e2e-tests-graph-node-1             | May 28 13:35:17.071 WARN GRAPH_POI_ACCESS_TOKEN not set; might leak POIs to the public via GraphQL
e2e-tests-graph-node-1             | May 28 13:35:17.071 INFO Reading configuration file `/etc/graphnode.toml`
e2e-tests-graph-node-1             | May 28 13:35:17.077 WARN No fork base URL specified, subgraph forking is disabled
e2e-tests-graph-node-1             | May 28 13:35:17.077 INFO Starting up
e2e-tests-graph-node-1             | May 28 13:35:17.077 INFO Trying IPFS node at: http://ipfs:5001/
e2e-tests-graph-node-1             | May 28 13:35:17.084 INFO Creating transport, capabilities: , url: http://host.docker.internal:3000/avalanche/default, provider: avalanche-default
e2e-tests-graph-node-1             | May 28 13:35:17.085 INFO Successfully connected to IPFS node at: http://ipfs:5001/
e2e-tests-graph-node-1             | May 28 13:35:17.088 INFO Creating transport, capabilities: , url: http://host.docker.internal:3000/mainnet/default, provider: mainnet-default
e2e-tests-graph-node-1             | May 28 13:35:17.119 INFO Creating transport, capabilities: archive, url: http://host.docker.internal:3000/mainnet/archive, provider: mainnet-archive
e2e-tests-graph-node-1             | May 28 13:35:17.244 INFO Creating transport, capabilities: archive, traces, url: http://host.docker.internal:3000/mainnet/archiveAndTraces, provider: mainnet-archive-traces
e2e-tests-graph-node-1             | May 28 13:35:17.953 DEBG Creating firehose networks [2 chains, ingestor default]
e2e-tests-graph-node-1             | May 28 13:35:17.953 INFO Configuring firehose endpoint, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:17.962 DEBG Creating firehose networks [2 chains, ingestor default]
e2e-tests-graph-node-1             | May 28 13:35:17.962 WARN Expensive queries file not set to a valid file: /etc/graph-node/expensive-queries.txt
e2e-tests-graph-node-1             | May 28 13:35:17.962 DEBG Cleaning up large notifications after about 300s, channel: store_events, component: NotificationListener
e2e-tests-graph-node-1             | May 28 13:35:17.971 INFO Connecting to Postgres, weight: 1, conn_pool_size: 10, url: postgresql://graph-node:HIDDEN_PASSWORD@postgres:5432/client__dev, pool: main, shard: primary
e2e-tests-graph-node-1             | May 28 13:35:17.972 INFO Pool successfully connected to Postgres, pool: main, shard: primary, component: Store
e2e-tests-graph-node-1             | May 28 13:35:17.975 INFO Connecting to Postgres, weight: 1, conn_pool_size: 10, url: postgresql://graph-node:HIDDEN_PASSWORD@postgres-second:5432/client__dev, pool: main, shard: second
e2e-tests-graph-node-1             | May 28 13:35:17.975 INFO Pool successfully connected to Postgres, pool: main, shard: second, component: Store
e2e-tests-graph-node-1             | May 28 13:35:17.975 DEBG Using postgres host order [Main], shard: primary, component: Store
e2e-tests-graph-node-1             | May 28 13:35:17.975 DEBG Using postgres host order [Main], shard: second, component: Store
e2e-tests-graph-node-1             | May 28 13:35:17.980 INFO Setting up fdw, pool: main, shard: primary, component: ConnectionPool
e2e-tests-graph-node-1             | May 28 13:35:17.982 INFO Setting up fdw, pool: main, shard: second, component: ConnectionPool
e2e-tests-graph-node-1             | May 28 13:35:17.993 INFO Running migrations, pool: main, shard: primary, component: ConnectionPool
e2e-tests-graph-node-1             | May 28 13:35:17.997 INFO Migrations finished, pool: main, shard: primary, component: ConnectionPool
e2e-tests-graph-node-1             | May 28 13:35:17.998 DEBG Release migration lock, pool: main, shard: primary, component: ConnectionPool
e2e-tests-graph-node-1             | May 28 13:35:18.000 DEBG Setup finished, setup_time_s: 0, pool: main, shard: primary, component: ConnectionPool
e2e-tests-graph-node-1             | May 28 13:35:18.003 INFO Running migrations, pool: main, shard: second, component: ConnectionPool
e2e-tests-graph-node-1             | May 28 13:35:18.003 INFO Migrations finished, pool: main, shard: second, component: ConnectionPool
e2e-tests-graph-node-1             | May 28 13:35:18.004 DEBG Release migration lock, pool: main, shard: second, component: ConnectionPool
e2e-tests-graph-node-1             | May 28 13:35:18.004 DEBG Setup finished, setup_time_s: 0, pool: main, shard: second, component: ConnectionPool
e2e-tests-graph-node-1             | May 28 13:35:18.004 DEBG Cleaning up large notifications after about 300s, channel: chain_head_updates, component: ChainHeadUpdateListener > NotificationListener
e2e-tests-graph-node-1             | May 28 13:35:18.012 INFO Connecting to Ethereum to get network identifier, capabilities: , provider: avalanche-default
e2e-tests-graph-node-1             | May 28 13:35:18.012 INFO Connecting to Ethereum to get network identifier, capabilities: , provider: mainnet-default
e2e-tests-graph-node-1             | May 28 13:35:18.014 INFO Connecting to Ethereum to get network identifier, capabilities: archive, provider: mainnet-archive
e2e-tests-graph-node-1             | May 28 13:35:18.014 INFO Connecting to Ethereum to get network identifier, capabilities: archive, traces, provider: mainnet-archive-traces
e2e-tests-graph-node-1             | May 28 13:35:18.115 INFO Connected to Ethereum, capabilities: , network_version: 43114, provider: avalanche-default
e2e-tests-graph-node-1             | May 28 13:35:18.123 INFO Connected to Ethereum, capabilities: , network_version: 1, provider: mainnet-default
e2e-tests-graph-node-1             | May 28 13:35:18.135 INFO Connected to Ethereum, capabilities: archive, traces, network_version: 1, provider: mainnet-archive-traces
e2e-tests-graph-node-1             | May 28 13:35:18.301 INFO Connected to Ethereum, capabilities: archive, network_version: 1, provider: mainnet-archive
e2e-tests-graph-node-1             | May 28 13:35:18.301 INFO Connecting to Firehose to get chain identifier, provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.301 INFO Requesting genesis block from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.301 DEBG Connecting to firehose to retrieve block for number 0, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.302 INFO Connecting to Firehose to get chain identifier, provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.302 INFO Requesting genesis block from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.302 DEBG Connecting to firehose to retrieve block for number 0, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.302 INFO Connecting to Firehose to get chain identifier, provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.302 INFO Requesting genesis block from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.302 DEBG Connecting to firehose to retrieve block for number 0, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.302 INFO Connecting to Firehose to get chain identifier, provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.302 INFO Requesting genesis block from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.302 DEBG Connecting to firehose to retrieve block for number 0, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.302 INFO Connecting to Firehose to get chain identifier, provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.302 INFO Requesting genesis block from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.302 DEBG Connecting to firehose to retrieve block for number 0, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.302 INFO Connecting to Firehose to get chain identifier, provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.302 INFO Requesting genesis block from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.302 DEBG Connecting to firehose to retrieve block for number 0, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.302 INFO Connecting to Firehose to get chain identifier, provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.302 INFO Requesting genesis block from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.302 DEBG Connecting to firehose to retrieve block for number 0, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.302 INFO Connecting to Firehose to get chain identifier, provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.302 INFO Requesting genesis block from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.302 DEBG Connecting to firehose to retrieve block for number 0, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.302 INFO Connecting to Firehose to get chain identifier, provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.302 INFO Requesting genesis block from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.302 DEBG Connecting to firehose to retrieve block for number 0, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.302 INFO Connecting to Firehose to get chain identifier, provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.302 INFO Requesting genesis block from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.302 DEBG Connecting to firehose to retrieve block for number 0, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.302 INFO Connecting to Firehose to get chain identifier, provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.303 INFO Requesting genesis block from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.303 DEBG Connecting to firehose to retrieve block for number 0, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.303 INFO Connecting to Firehose to get chain identifier, provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.303 INFO Requesting genesis block from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.303 DEBG Connecting to firehose to retrieve block for number 0, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.303 INFO Connecting to Firehose to get chain identifier, provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.303 INFO Requesting genesis block from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.303 DEBG Connecting to firehose to retrieve block for number 0, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.303 INFO Connecting to Firehose to get chain identifier, provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.303 INFO Requesting genesis block from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.303 DEBG Connecting to firehose to retrieve block for number 0, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.303 INFO Connecting to Firehose to get chain identifier, provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.303 INFO Requesting genesis block from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.303 DEBG Connecting to firehose to retrieve block for number 0, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.303 INFO Connecting to Firehose to get chain identifier, provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.303 INFO Requesting genesis block from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.303 DEBG Connecting to firehose to retrieve block for number 0, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.303 INFO Connecting to Firehose to get chain identifier, provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.303 INFO Requesting genesis block from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.303 DEBG Connecting to firehose to retrieve block for number 0, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.303 INFO Connecting to Firehose to get chain identifier, provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.303 INFO Requesting genesis block from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.303 DEBG Connecting to firehose to retrieve block for number 0, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.303 INFO Connecting to Firehose to get chain identifier, provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.303 INFO Requesting genesis block from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.303 DEBG Connecting to firehose to retrieve block for number 0, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.303 INFO Connecting to Firehose to get chain identifier, provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.303 INFO Requesting genesis block from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.303 DEBG Connecting to firehose to retrieve block for number 0, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.554 DEBG Retrieving block(s) from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.557 DEBG Retrieving block(s) from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.562 DEBG Retrieving block(s) from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.566 DEBG Retrieving block(s) from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.566 DEBG Retrieving block(s) from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.576 DEBG Retrieving block(s) from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.582 DEBG Retrieving block(s) from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.586 DEBG Retrieving block(s) from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.598 DEBG Retrieving block(s) from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.600 DEBG Retrieving block(s) from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.606 DEBG Retrieving block(s) from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.609 DEBG Retrieving block(s) from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.609 DEBG Retrieving block(s) from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.625 DEBG Retrieving block(s) from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.633 DEBG Retrieving block(s) from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.634 DEBG Retrieving block(s) from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.639 DEBG Retrieving block(s) from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.686 DEBG Retrieving block(s) from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:18.733 DEBG Retrieving block(s) from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:19.188 DEBG Retrieving block(s) from firehose, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:19.949 INFO Connected to Firehose, genesis_block: #0 (d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3), provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:20.129 INFO Connected to Firehose, genesis_block: #0 (d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3), provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:20.402 INFO Connected to Firehose, genesis_block: #0 (d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3), provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:20.595 INFO Connected to Firehose, genesis_block: #0 (d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3), provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:20.669 INFO Connected to Firehose, genesis_block: #0 (d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3), provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:20.670 INFO Connected to Firehose, genesis_block: #0 (d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3), provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:20.865 INFO Connected to Firehose, genesis_block: #0 (d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3), provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:21.030 INFO Connected to Firehose, genesis_block: #0 (d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3), provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:21.354 INFO Connected to Firehose, genesis_block: #0 (d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3), provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:21.395 INFO Connected to Firehose, genesis_block: #0 (d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3), provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:21.535 INFO Connected to Firehose, genesis_block: #0 (d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3), provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:21.535 INFO Connected to Firehose, genesis_block: #0 (d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3), provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:21.573 INFO Connected to Firehose, genesis_block: #0 (d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3), provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:22.007 INFO Connected to Firehose, genesis_block: #0 (d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3), provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:22.014 INFO Connected to Firehose, genesis_block: #0 (d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3), provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:22.258 INFO Connected to Firehose, genesis_block: #0 (d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3), provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:22.318 INFO Connected to Firehose, genesis_block: #0 (d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3), provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:22.858 INFO Connected to Firehose, genesis_block: #0 (d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3), provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:22.875 INFO Connected to Firehose, genesis_block: #0 (d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3), provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:25.340 INFO Connected to Firehose, genesis_block: #0 (d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3), provider: streamingfast-firehose-mainnet, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 28 13:35:25.342 ERRO the net version for chain mainnet has changed from 1 to 0 since the last time we ran, component: BlockStore
e2e-tests-graph-node-1             | May 28 13:35:25.361 INFO Creating LoadManager in disabled mode, component: LoadManager
e2e-tests-graph-node-1             | May 28 13:35:25.361 INFO Started block ingestor, network_name: avalanche
e2e-tests-graph-node-1             | May 28 13:35:25.361 INFO Started block ingestor, network_name: mainnet
e2e-tests-graph-node-1             | May 28 13:35:25.361 INFO Starting firehose block ingestor for network, network_name: avalanche
e2e-tests-graph-node-1             | May 28 13:35:25.362 INFO Starting firehose block ingestor for network, network_name: mainnet
e2e-tests-graph-node-1             | May 28 13:35:25.362 INFO Starting job runner with 5 jobs, component: JobRunner
e2e-tests-graph-node-1             | May 28 13:35:25.362 DEBG Schedule for Vacuum subgraphs.subgraph_deployment, first_run_in_s: 0, interval_s: 60, component: JobRunner
e2e-tests-graph-node-1             | May 28 13:35:25.362 DEBG Schedule for Report pg_notification_queue_usage(), first_run_in_s: 0, interval_s: 60, component: JobRunner
e2e-tests-graph-node-1             | May 28 13:35:25.362 DEBG Schedule for Reconcile important tables from the primary, first_run_in_s: 9, interval_s: 900, component: JobRunner
e2e-tests-graph-node-1             | May 28 13:35:25.362 DEBG Schedule for Record and remove unused deployments, first_run_in_s: 79, interval_s: 7200, component: JobRunner
e2e-tests-graph-node-1             | May 28 13:35:25.362 DEBG Schedule for Refresh materialized views, first_run_in_s: 237, interval_s: 21600, component: JobRunner
e2e-tests-graph-node-1             | May 28 13:35:25.365 INFO Starting JSON-RPC admin server at: http://localhost:8020, component: JsonRpcServer
e2e-tests-graph-node-1             | May 28 13:35:25.365 INFO Starting GraphQL HTTP server at: http://localhost:8000, component: GraphQLServer
e2e-tests-graph-node-1             | May 28 13:35:25.365 INFO Starting index node server at: http://localhost:8030, component: IndexNodeServer
e2e-tests-graph-node-1             | May 28 13:35:25.365 INFO Starting metrics server at: http://localhost:8040, component: MetricsServer
e2e-tests-graph-node-1             | May 28 13:35:25.365 INFO Starting GraphQL WebSocket server at: ws://localhost:8001, component: SubscriptionServer
e2e-tests-graph-node-1             | May 28 13:35:25.365 INFO Started all assigned subgraphs, node_id: default, count: 0, component: SubgraphRegistrar
e2e-tests-graph-node-1             | May 28 13:35:25.366 INFO Trying to reconnect the Blockstream after disconnect, cursor: , endpoint uri: streamingfast-firehose-mainnet, network_name: mainnet, provider: streamingfast-firehose-mainnet, component: EthereumFirehoseBlockIngestor
e2e-tests-graph-node-1             | May 28 13:35:25.437 INFO Downloading latest blocks from Ethereum, this may take a few minutes..., provider: avalanche-default, component: EthereumPollingBlockIngestor
e2e-tests-graph-node-1             | May 28 13:35:25.501 INFO Blockstream connected, consuming blocks, network_name: mainnet, provider: streamingfast-firehose-mainnet, component: EthereumFirehoseBlockIngestor

IPFS hash

No response

Subgraph name or link to explorer

No response

Some information to help us out

  • Tick this box if this bug is caused by a regression found in the latest release.
  • Tick this box if this bug is specific to the hosted service.
  • I have searched the issue tracker to make sure this issue is not a duplicate.

OS information

None
@paymog paymog added the bug Something isn't working label May 28, 2023
@paymog
Copy link
Author

paymog commented May 28, 2023

This panic does not happen if I disable the firehose provider.

@paymog
Copy link
Author

paymog commented May 29, 2023

This panic doesn't happen on 0.30.0, but a different panic happens instead and this one loops whereas the one above resolves after the first crash

e2e-tests-graph-node-1             | May 29 08:47:57.827 INFO Graph Node version: 0.30.0 (2023-05-28)
e2e-tests-graph-node-1             | May 29 08:47:57.827 WARN GRAPH_POI_ACCESS_TOKEN not set; might leak POIs to the public via GraphQL
e2e-tests-graph-node-1             | May 29 08:47:57.827 INFO Reading configuration file `/etc/graphnode.toml`
e2e-tests-graph-node-1             | May 29 08:47:57.839 WARN No fork base URL specified, subgraph forking is disabled
e2e-tests-graph-node-1             | May 29 08:47:57.840 INFO Starting up
e2e-tests-graph-node-1             | May 29 08:47:57.840 INFO Trying IPFS node at: http://ipfs:5001/
e2e-tests-graph-node-1             | May 29 08:47:57.853 INFO Creating transport, capabilities: , url: http://host.docker.internal:3000/avalanche/default, provider: avalanche-default
e2e-tests-graph-node-1             | May 29 08:47:57.853 INFO Successfully connected to IPFS node at: http://ipfs:5001/
e2e-tests-graph-node-1             | May 29 08:47:57.861 INFO Creating transport, capabilities: , url: http://host.docker.internal:3000/mainnet/default, provider: mainnet-default
e2e-tests-graph-node-1             | May 29 08:47:58.415 INFO Creating transport, capabilities: archive, url: http://host.docker.internal:3000/mainnet/archive, provider: mainnet-archive
e2e-tests-graph-node-1             | May 29 08:47:58.806 INFO Creating transport, capabilities: archive, traces, url: http://host.docker.internal:3000/mainnet/archiveAndTraces, provider: mainnet-archive-traces
e2e-tests-graph-node-1             | May 29 08:47:59.520 DEBG Creating firehose networks [2 chains, ingestor default]
e2e-tests-graph-node-1             | May 29 08:47:59.520 INFO Configuring firehose endpoint, provider: streamingfast-firehose-mainnet
e2e-tests-graph-node-1             | May 29 08:47:59.530 DEBG Creating firehose networks [2 chains, ingestor default]
e2e-tests-graph-node-1             | May 29 08:47:59.531 WARN Expensive queries file not set to a valid file: /etc/graph-node/expensive-queries.txt
e2e-tests-graph-node-1             | May 29 08:47:59.532 DEBG Cleaning up large notifications after about 300s, channel: store_events, component: NotificationListener
e2e-tests-graph-node-1             | May 29 08:47:59.543 INFO Connecting to Postgres, weight: 1, conn_pool_size: 10, url: postgresql://graph-node:HIDDEN_PASSWORD@postgres:5432/client__dev, pool: main, shard: primary
e2e-tests-graph-node-1             | May 29 08:47:59.545 INFO Pool successfully connected to Postgres, pool: main, shard: primary, component: Store
e2e-tests-graph-node-1             | May 29 08:47:59.548 INFO Connecting to Postgres, weight: 1, conn_pool_size: 10, url: postgresql://graph-node:HIDDEN_PASSWORD@postgres-second:5432/client__dev, pool: main, shard: second
e2e-tests-graph-node-1             | May 29 08:47:59.550 INFO Pool successfully connected to Postgres, pool: main, shard: second, component: Store
e2e-tests-graph-node-1             | May 29 08:47:59.551 DEBG Using postgres host order [Main], shard: primary, component: Store
e2e-tests-graph-node-1             | May 29 08:47:59.553 DEBG Using postgres host order [Main], shard: second, component: Store
e2e-tests-graph-node-1             | May 29 08:47:59.561 INFO Setting up fdw, pool: main, shard: primary, component: ConnectionPool
e2e-tests-graph-node-1             | May 29 08:47:59.567 INFO Setting up fdw, pool: main, shard: second, component: ConnectionPool
e2e-tests-graph-node-1             | May 29 08:47:59.579 INFO Running migrations, pool: main, shard: second, component: ConnectionPool
e2e-tests-graph-node-1             | May 29 08:47:59.580 INFO Running migrations, pool: main, shard: primary, component: ConnectionPool
e2e-tests-graph-node-1             | May 29 08:48:00.202 INFO Migrations finished, pool: main, shard: second, component: ConnectionPool
e2e-tests-graph-node-1             | May 29 08:48:00.203 DEBG Postgres migration output, output: Running migration 20180710061642 Running migration 20180710061659 Running migration 20180710062717 Running migration 20180710062730 Running migration 20180816143800 Running migration 20180822130000 Running migration 20180822140000 Running migration 20180829120000 Running migration 20180907220000 Running migration 20180918180000 Running migration 20181202114100 Running migration 20181203150000 Running migration 20181221003727 Running migration 20190107120000 Running migration 20190118195346 Running migration 20190118195526 Running migration 20190201045927 Running migration 20190204114900 Running migration 20190225182843 Running migration 20190226182914 Running migration 20190226183156 Running migration 20190227035443 Running migration 20190228190800 Running migration 20190304235349 Running migration 20190307171355 Running migration 20190328004319 Running migration 20190331010824 Running migration 20190419171709 Running migration 20190422190022 Running migration 20190503164052 Running migration 20190509135900 Running migration 20190509232642 Running migration 20190514200255 Running migration 20190515215022 Running migration 20190605214320 Running migration 20190626164405 Running migration 20190720195916 Running migration 20190802001120 Running migration 20190901135850 Running migration 20190902230613 Running migration 20191001173616 Running migration 20200114235608 Running migration 20200117013633 Running migration 20200124065338 Running migration 20200306020253 Running migration 20200311162100 Running migration 20200325170527 Running migration 20200404002817 Running migration 20200410111111 Running migration 20200516225611 Running migration 20200707002933 Running migration 20200731162138 Running migration 20201031150000 Running migration 20201103170839 Running migration 20201110100000 Running migration 20201211142000 Running migration 20201212000001 Running migration 20201212000002 Running migration 20201212000003 Running migration 20201212000004 Running migration 20201212000005 Running migration 20201215000000 Running migration 2020127190800 Running migration 20210107004939 Running migration 20210114175654 Running migration 20210114193022 Running migration 20210115013524 Running migration 20210119033749 Running migration 20210126044443 Running migration 20210126172953 Running migration 20210126173710 Running migration 20210126204036 Running migration 20210217205502 Running migration 20210218000721 Running migration 20210218171423 Running migration 20210224051050 Running migration 20210225233156 Running migration 20210311010830 Running migration 20210311231340 Running migration 20210312014815 Running migration 20210312070453 Running migration 20210316001809 Running migration 20210316165131 Running migration 20210319161012 Running migration 20210320001347 Running migration 20210324220541 Running migration 20210520130220 Running migration 20210521165219 Running migration 20210601222649 Running migration 20210614201635 Running migration 20210709144144 Running migration 20211202212506 Running migration 20211205061157 Running migration 2022030907000 Running migration 20220321172028 Running migration 20220323010300 Running migration 20220426125552 Running migration 20220612003442 Running migration 20220919161239 Running migration 20221013000000 Running migration 20221103213140 Running migration 20221110185105 Running migration 20230124192319, pool: main, shard: second, component: ConnectionPool
e2e-tests-graph-node-1             | May 29 08:48:00.204 INFO Mapping metadata from second, pool: main, shard: primary, component: ConnectionPool
e2e-tests-graph-node-1             | May 29 08:48:00.233 INFO Migrations finished, pool: main, shard: primary, component: ConnectionPool
e2e-tests-graph-node-1             | May 29 08:48:00.234 DEBG Postgres migration output, output: Running migration 20180710061642 Running migration 20180710061659 Running migration 20180710062717 Running migration 20180710062730 Running migration 20180816143800 Running migration 20180822130000 Running migration 20180822140000 Running migration 20180829120000 Running migration 20180907220000 Running migration 20180918180000 Running migration 20181202114100 Running migration 20181203150000 Running migration 20181221003727 Running migration 20190107120000 Running migration 20190118195346 Running migration 20190118195526 Running migration 20190201045927 Running migration 20190204114900 Running migration 20190225182843 Running migration 20190226182914 Running migration 20190226183156 Running migration 20190227035443 Running migration 20190228190800 Running migration 20190304235349 Running migration 20190307171355 Running migration 20190328004319 Running migration 20190331010824 Running migration 20190419171709 Running migration 20190422190022 Running migration 20190503164052 Running migration 20190509135900 Running migration 20190509232642 Running migration 20190514200255 Running migration 20190515215022 Running migration 20190605214320 Running migration 20190626164405 Running migration 20190720195916 Running migration 20190802001120 Running migration 20190901135850 Running migration 20190902230613 Running migration 20191001173616 Running migration 20200114235608 Running migration 20200117013633 Running migration 20200124065338 Running migration 20200306020253 Running migration 20200311162100 Running migration 20200325170527 Running migration 20200404002817 Running migration 20200410111111 Running migration 20200516225611 Running migration 20200707002933 Running migration 20200731162138 Running migration 20201031150000 Running migration 20201103170839 Running migration 20201110100000 Running migration 20201211142000 Running migration 20201212000001 Running migration 20201212000002 Running migration 20201212000003 Running migration 20201212000004 Running migration 20201212000005 Running migration 20201215000000 Running migration 2020127190800 Running migration 20210107004939 Running migration 20210114175654 Running migration 20210114193022 Running migration 20210115013524 Running migration 20210119033749 Running migration 20210126044443 Running migration 20210126172953 Running migration 20210126173710 Running migration 20210126204036 Running migration 20210217205502 Running migration 20210218000721 Running migration 20210218171423 Running migration 20210224051050 Running migration 20210225233156 Running migration 20210311010830 Running migration 20210311231340 Running migration 20210312014815 Running migration 20210312070453 Running migration 20210316001809 Running migration 20210316165131 Running migration 20210319161012 Running migration 20210320001347 Running migration 20210324220541 Running migration 20210520130220 Running migration 20210521165219 Running migration 20210601222649 Running migration 20210614201635 Running migration 20210709144144 Running migration 20211202212506 Running migration 20211205061157 Running migration 2022030907000 Running migration 20220321172028 Running migration 20220323010300 Running migration 20220426125552 Running migration 20220612003442 Running migration 20220919161239 Running migration 20221013000000 Running migration 20221103213140 Running migration 20221110185105 Running migration 20230124192319, pool: main, shard: primary, component: ConnectionPool
e2e-tests-graph-node-1             | May 29 08:48:00.296 DEBG Release migration lock, pool: main, shard: second, component: ConnectionPool
e2e-tests-graph-node-1             | May 29 08:48:00.297 INFO Mapping primary, pool: main, shard: primary, component: ConnectionPool
e2e-tests-graph-node-1             | May 29 08:48:00.298 DEBG Setup finished, setup_time_s: 0, pool: main, shard: second, component: ConnectionPool
e2e-tests-graph-node-1             | May 29 08:48:00.299 INFO Mapping primary, pool: main, shard: second, component: ConnectionPool
e2e-tests-graph-node-1             | May 29 08:48:00.334 INFO Mapping metadata from primary, pool: main, shard: second, component: ConnectionPool
e2e-tests-graph-node-1             | May 29 08:48:00.386 DEBG Release migration lock, pool: main, shard: primary, component: ConnectionPool
e2e-tests-graph-node-1             | May 29 08:48:00.387 DEBG Setup finished, setup_time_s: 0, pool: main, shard: primary, component: ConnectionPool
e2e-tests-graph-node-1             | May 29 08:48:00.387 DEBG Cleaning up large notifications after about 300s, channel: chain_head_updates, component: ChainHeadUpdateListener > NotificationListener
e2e-tests-graph-node-1             | May 29 08:48:00.395 INFO Connecting to Ethereum to get network identifier, capabilities: , provider: mainnet-default
e2e-tests-graph-node-1             | May 29 08:48:00.395 INFO Connecting to Ethereum to get network identifier, capabilities: archive, provider: mainnet-archive
e2e-tests-graph-node-1             | May 29 08:48:00.396 INFO Connecting to Ethereum to get network identifier, capabilities: archive, traces, provider: mainnet-archive-traces
e2e-tests-graph-node-1             | May 29 08:48:00.397 INFO Connecting to Ethereum to get network identifier, capabilities: , provider: avalanche-default
e2e-tests-graph-node-1             | May 29 08:48:00.534 INFO Connected to Ethereum, capabilities: archive, network_version: 1, provider: mainnet-archive
e2e-tests-graph-node-1             | May 29 08:48:00.559 INFO Connected to Ethereum, capabilities: , network_version: 43114, provider: avalanche-default
e2e-tests-graph-node-1             | May 29 08:48:00.561 INFO Connected to Ethereum, capabilities: , network_version: 1, provider: mainnet-default
e2e-tests-graph-node-1             | May 29 08:48:00.568 INFO Connected to Ethereum, capabilities: archive, traces, network_version: 1, provider: mainnet-archive-traces
e2e-tests-graph-node-1             | May 29 08:48:00.612 INFO Creating LoadManager in disabled mode, component: LoadManager
e2e-tests-graph-node-1             | May 29 08:48:00.612 INFO Starting block ingestors with 1 chains [avalanche]
e2e-tests-graph-node-1             | May 29 08:48:00.612 INFO Starting block ingestor for network, network_name: avalanche
e2e-tests-graph-node-1             | May 29 08:48:00.612 INFO Starting firehose block ingestors with 1 chains [mainnet]
e2e-tests-graph-node-1             | May 29 08:48:00.612 INFO Starting firehose block ingestor for network, network_name: mainnet
e2e-tests-graph-node-1             | thread 'tokio-runtime-worker' panicked at 'One Firehose endpoint should exist at that execution point: all connections saturated with 100 connections, increase the firehose conn_pool_size or limit for the node', node/src/main.rs:970:18
e2e-tests-graph-node-1             | note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
e2e-tests-graph-node-1             | Panic in tokio task, aborting!
e2e-tests-graph-node-1             | Aborted
e2e-tests-graph-node-1 exited with code 134

@leoyvens
Copy link
Collaborator

leoyvens commented Jun 5, 2023

I was able to reproduce the Creating the BlockStore works: Unknown(duplicate key value violates unique constraint "chains_name_key") with a config as simple as:

[store]
[store.primary]
connection = "<DB_URL>"
pool_size = 10

[chains]
ingestor = "default"

[deployment]
[[deployment.rule]]
shard = ["primary"]
indexers = [ "default" ]

[chains.mainnet]
shard = "primary"
protocol = "ethereum"
provider = [
  { label = "mainnet", details = { type = "web3", url = "<RPC_URL>", features = [] }},
  { label = "mainnet-firehose-sf", details = { type = "firehose", url = "<FIREHOSE_URL>", token = "<FIREHOSE TOKEN>", features = ["compression", "filters"] }},
]

@neysofu
Copy link
Member

neysofu commented Jun 7, 2023

Thanks for the repro @leoyvens! I bisected it down to #4553.

@leoyvens leoyvens mentioned this issue Jun 7, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix panic on first startup when mixing firehose and rpc graphprotocol/graph-node
3 participants
@paymog @leoyvens @neysofu