Skip to content
This repository has been archived by the owner on Feb 6, 2025. It is now read-only.

Using snapshot, running the node gives this error. #85

Closed
1 task done
greyireland opened this issue Jul 29, 2024 · 15 comments
Closed
1 task done

Using snapshot, running the node gives this error. #85

greyireland opened this issue Jul 29, 2024 · 15 comments
Assignees

Comments

@greyireland
Copy link

greyireland commented Jul 29, 2024

Describe the bug

Error: database integrity error occurred: trying to append data to Sidecars as block #40888244 but expected block #40888243

Caused by:
trying to append data to Sidecars as block #40888244 but expected block #40888243

Location:
/project/crates/node/builder/src/launch/mod.rs:411:28

Steps to reproduce

Download this:https://github.com/fuzzland/snapshots
Run:./bsc-reth node --datadir ~/reth/bsc --http --http.api all --http.addr 0.0.0.0 --http.port 8547 --ws --ws.api all --ws.addr 0.0.0.0 --ws.port 8548 --metrics 0.0.0.0:9001

Node logs

-

Platform(s)

Linux (x86)

What version/commit are you on?

./reth --version
reth Version: 1.0.1
Commit SHA: 4d9ea56
Build Timestamp: 2024-07-18T07:18:40.601597195Z
Build Features: bsc,jemalloc
Build Profile: maxperf

What database version are you on?

Which chain / network are you on?

bsc

What type of node are you running?

Archive (default)

What prune config do you use, if any?

If you've built Reth from source, provide the full command you used

./bsc-reth node --datadir ~/reth/bsc --http --http.api all --http.addr 0.0.0.0 --http.port 8547 --ws --ws.api all --ws.addr 0.0.0.0 --ws.port 8548 --metrics 0.0.0.0:9001

Code of Conduct

  • I agree to follow the Code of Conduct
@pythonberg1997
Copy link
Contributor

pythonberg1997 commented Jul 31, 2024

This issue has been fixed in #89. Please update the client and retry.

@greyireland
Copy link
Author

Similar error log
Reth version:Starting reth version="1.0.1 (c3361c9)

16|reth  | 2024-08-04T00:07:39.413788Z ERROR Stage encountered a fatal error: database integrity error occurred: trying to append data to Transactions as block #41053633 but expected block #41053634 stage=Bodies
16|reth  | 2024-08-04T00:07:39.414084Z ERROR shutting down due to error
16|reth  | Error: database integrity error occurred: trying to append data to Transactions as block #41053633 but expected block #41053634
16|reth  | Caused by:
16|reth  |     trying to append data to Transactions as block #41053633 but expected block #41053634
16|reth  | Location:
16|reth  |     /project/crates/node/builder/src/launch/mod.rs:411:28

@greyireland
Copy link
Author

Error: database integrity error occurred: trying to append data to Transactions as block #41052678 but expected block #41052679

Caused by:
    trying to append data to Transactions as block #41052678 but expected block #41052679

Location:
    /project/crates/node/builder/src/launch/mod.rs:411:28
Error: database integrity error occurred: trying to append data to Transactions as block #41053066 but expected block #41053067

Caused by:
    trying to append data to Transactions as block #41053066 but expected block #41053067

Location:
    /project/crates/node/builder/src/launch/mod.rs:411:28
Error: database integrity error occurred: trying to append data to Transactions as block #41053371 but expected block #41053372

Caused by:
    trying to append data to Transactions as block #41053371 but expected block #41053372

Location:
    /project/crates/node/builder/src/launch/mod.rs:411:28
Error: database integrity error occurred: trying to append data to Transactions as block #41053633 but expected block #41053634

Caused by:
    trying to append data to Transactions as block #41053633 but expected block #41053634

Location:
    /project/crates/node/builder/src/launch/mod.rs:411:28

@unclezoro
Copy link
Collaborator

@pythonberg1997 do we get a chance to try the snapshot?

@pythonberg1997
Copy link
Contributor

pythonberg1997 commented Aug 5, 2024

Could you provide more logs in debug level? Especially before the node exit. @greyireland

@greyireland
Copy link
Author

I am not using the snapshots above, I am using a synchronized full node.
cmd

RUST_LOG=debug exec ./reth node --full --datadir /home/ubuntu/reth/bsc --datadir.static_files /home/ubuntu/data/rethdata/static_files --http --http.api all --http.addr 127.0.0.1 --http.port 8547 --ws --ws.api all --ws.addr 127.0.0.1 --ws.port 8548 --metrics 0.0.0.0:9001

debug logs

2024-08-05T09:17:27.782163Z  INFO reth::cli: Initialized tracing, debug log directory: /home/ubuntu/.cache/reth/logs/bsc
2024-08-05T09:17:27.785039Z  INFO reth::cli: Starting reth version="1.0.1 (c3361c97)"
2024-08-05T09:17:27.785653Z  INFO reth::cli: Opening database path="/home/ubuntu/reth/bsc/db"
2024-08-05T09:17:27.826185Z DEBUG reth_node_builder::launch::common: Raised file descriptor limit from=1048576 to=1048576
2024-08-05T09:17:27.830875Z  INFO reth::cli: Configuration loaded path="/home/ubuntu/reth/bsc/reth.toml"
2024-08-05T09:17:27.842510Z  INFO reth::cli: Verifying storage consistency.
2024-08-05T09:17:27.845891Z  INFO reth::cli: Database opened
2024-08-05T09:17:27.845903Z  INFO reth::cli: Starting metrics endpoint addr=0.0.0.0:9001
2024-08-05T09:17:27.845983Z DEBUG reth::cli: Initializing genesis chain=bsc genesis=0x0d21840abff46b96c84b2ac9e10e4f5cdaeb5693cb665db62a2f3b02d2d57b5b
2024-08-05T09:17:27.846034Z DEBUG reth_db_common::init: Genesis already written, skipping.
2024-08-05T09:17:27.846043Z  INFO reth::cli:
Pre-merge hard forks (block based):
- Frontier                         @0
- Homestead                        @0
- Tangerine                        @0
- SpuriousDragon                   @0
- Byzantium                        @0
- Constantinople                   @0
- Petersburg                       @0
- Istanbul                         @0
- MuirGlacier                      @0
- Ramanujan                        @0
- Niels                            @0
- MirrorSync                       @5184000
- Bruno                            @13082000
- Euler                            @18907621
- Nano                             @21962149
- Moran                            @22107423
- Gibbs                            @23846001
- Planck                           @27281024
- Luban                            @29020050
- Plato                            @30720096
- Berlin                           @31302048
- London                           @31302048
- Hertz                            @31302048
- HertzFix                         @34140700
Post-merge hard forks (timestamp based):
- Shanghai                         @1705996800
- Kepler                           @1705996800
- Feynman                          @1713419340
- FeynmanFix                       @1713419340
- Cancun                           @1718863500
- Haber                            @1718863500
2024-08-05T09:17:27.846068Z DEBUG reth::cli: Spawning stages metrics listener task
2024-08-05T09:17:27.846118Z DEBUG reth::cli: creating components
2024-08-05T09:17:27.846184Z DEBUG txpool::blob: Removed blob store directory blob_dir="/home/ubuntu/reth/bsc/blobstore"
2024-08-05T09:17:27.846192Z DEBUG txpool::blob: Creating blob store blob_dir="/home/ubuntu/reth/bsc/blobstore"
2024-08-05T09:17:27.846367Z  INFO reth::cli: Transaction pool initialized
2024-08-05T09:17:27.846386Z DEBUG reth::cli: Spawned txpool maintenance task
2024-08-05T09:17:27.850945Z  INFO net::peers: Loading saved peers file=/home/ubuntu/reth/bsc/known-peers.json
2024-08-05T09:17:27.851861Z DEBUG discv4: pinging boot node record=NodeRecord { address: 52.69.102.73, tcp_port: 30311, udp_port: 30311, id: 0x433c8bfdf53a3e2268ccb1b829e47f629793291cbddf0c76ae626da802f90532251fc558e2e0d10d6725e759088439bf1cd4714716b03a259a35d4b2e4acfa7f }
2024-08-05T09:17:27.851954Z DEBUG discv4: pinging boot node record=NodeRecord { address: 18.203.152.54, tcp_port: 30311, udp_port: 30311, id: 0xfac42fb0ba082b7d1eebded216db42161163d42e4f52c9e47716946d64468a62da4ba0b1cac0df5e8bf1e5284861d757339751c33d51dfef318be5168803d0b5 }
2024-08-05T09:17:27.851992Z DEBUG discv4: pinging boot node record=NodeRecord { address: 34.250.32.100, tcp_port: 30311, udp_port: 30311, id: 0x3063d1c9e1b824cfbb7c7b6abafa34faec6bb4e7e06941d218d760acdd7963b274278c5c3e63914bd6d1b58504c59ec5522c56f883baceb8538674b92da48a96 }
2024-08-05T09:17:27.852026Z DEBUG discv4: pinging boot node record=NodeRecord { address: 35.73.84.120, tcp_port: 30311, udp_port: 30311, id: 0x571bee8fb902a625942f10a770ccf727ae2ba1bab2a2b64e121594a99c9437317f6166a395670a00b7d93647eacafe598b6bbcef15b40b6d1a10243865a3e80f }
2024-08-05T09:17:27.852065Z DEBUG discv4: pinging boot node record=NodeRecord { address: 34.204.214.24, tcp_port: 30311, udp_port: 30311, id: 0xad78c64a4ade83692488aa42e4c94084516e555d3f340d9802c2bf106a3df8868bc46eae083d2de4018f40e8d9a9952c32a0943cd68855a9bc9fd07aac982a6d }
2024-08-05T09:17:27.852100Z DEBUG discv4: pinging boot node record=NodeRecord { address: 107.20.191.97, tcp_port: 30311, udp_port: 30311, id: 0x5db798deb67df75d073f8e2953dad283148133acb520625ea804c9c4ad09a35f13592a762d8f89056248f3889f6dcc33490c145774ea4ff2966982294909b37a }
2024-08-05T09:17:27.852872Z DEBUG reth::cli: configured blockchain tree
2024-08-05T09:17:27.852882Z DEBUG reth::cli: calling on_component_initialized hook
2024-08-05T09:17:27.853219Z  INFO reth::cli: StaticFileProducer initialized
2024-08-05T09:17:27.853505Z DEBUG discv4: pinging boot node record=NodeRecord { address: 52.69.102.73, tcp_port: 30311, udp_port: 30311, id: 0x433c8bfdf53a3e2268ccb1b829e47f629793291cbddf0c76ae626da802f90532251fc558e2e0d10d6725e759088439bf1cd4714716b03a259a35d4b2e4acfa7f }
2024-08-05T09:17:27.853546Z DEBUG discv4: pinging boot node record=NodeRecord { address: 18.203.152.54, tcp_port: 30311, udp_port: 30311, id: 0xfac42fb0ba082b7d1eebded216db42161163d42e4f52c9e47716946d64468a62da4ba0b1cac0df5e8bf1e5284861d757339751c33d51dfef318be5168803d0b5 }
2024-08-05T09:17:27.853552Z DEBUG discv4: pinging boot node record=NodeRecord { address: 34.250.32.100, tcp_port: 30311, udp_port: 30311, id: 0x3063d1c9e1b824cfbb7c7b6abafa34faec6bb4e7e06941d218d760acdd7963b274278c5c3e63914bd6d1b58504c59ec5522c56f883baceb8538674b92da48a96 }
2024-08-05T09:17:27.853555Z DEBUG discv4: pinging boot node record=NodeRecord { address: 35.73.84.120, tcp_port: 30311, udp_port: 30311, id: 0x571bee8fb902a625942f10a770ccf727ae2ba1bab2a2b64e121594a99c9437317f6166a395670a00b7d93647eacafe598b6bbcef15b40b6d1a10243865a3e80f }
2024-08-05T09:17:27.853559Z DEBUG discv4: pinging boot node record=NodeRecord { address: 34.204.214.24, tcp_port: 30311, udp_port: 30311, id: 0xad78c64a4ade83692488aa42e4c94084516e555d3f340d9802c2bf106a3df8868bc46eae083d2de4018f40e8d9a9952c32a0943cd68855a9bc9fd07aac982a6d }
2024-08-05T09:17:27.853564Z DEBUG discv4: pinging boot node record=NodeRecord { address: 107.20.191.97, tcp_port: 30311, udp_port: 30311, id: 0x5db798deb67df75d073f8e2953dad283148133acb520625ea804c9c4ad09a35f13592a762d8f89056248f3889f6dcc33490c145774ea4ff2966982294909b37a }
2024-08-05T09:17:27.853654Z DEBUG reqwest::connect: starting new connection: http://ipinfo.io/
2024-08-05T09:17:27.853732Z DEBUG reqwest::connect: starting new connection: http://icanhazip.com/
2024-08-05T09:17:27.853800Z DEBUG reqwest::connect: starting new connection: http://ifconfig.me/
2024-08-05T09:17:27.853890Z DEBUG hyper_util::client::legacy::connect::dns: resolving host="ipinfo.io"
2024-08-05T09:17:27.854017Z DEBUG hyper_util::client::legacy::connect::dns: resolving host="icanhazip.com"
2024-08-05T09:17:27.854061Z DEBUG hyper_util::client::legacy::connect::dns: resolving host="ifconfig.me"
2024-08-05T09:17:27.854891Z DEBUG hyper_util::client::legacy::connect::http: connecting to 104.16.185.241:80
2024-08-05T09:17:27.855210Z DEBUG hyper_util::client::legacy::connect::http: connecting to 34.117.59.81:80
2024-08-05T09:17:27.855247Z DEBUG hyper_util::client::legacy::connect::http: connecting to 34.160.111.145:80
2024-08-05T09:17:27.855819Z DEBUG hyper_util::client::legacy::connect::http: connected to 104.16.185.241:80
2024-08-05T09:17:27.856363Z DEBUG hyper_util::client::legacy::connect::http: connected to 34.117.59.81:80
2024-08-05T09:17:27.856624Z DEBUG hyper_util::client::legacy::connect::http: connected to 34.160.111.145:80
2024-08-05T09:17:27.864957Z DEBUG hyper_util::client::legacy::pool: pooling idle connection for ("http", icanhazip.com)
2024-08-05T09:17:27.865007Z DEBUG discv4: Updating external ip external_ip=103.106.59.25
2024-08-05T09:17:27.865178Z DEBUG discv4: Updated local ENR enr=Enr { id: Some("v4"), seq: 2, NodeId: 0x9576166c8569842ba094ace22087fee14ad691efed63cd3bb4ecb1e87c653e71, signature: "eb955bb1d1c8d1b16d4055d00a3c510b621bc6f5390cf0304b29c106d42057b63becb65fddb526d1e0cb1591e31b6f6d1207405c0f726cdf3cb60779b18a8ec0", IpV4 UDP Socket: Some(103.106.59.25:30303), IpV6 UDP Socket: None, IpV4 TCP Socket: Some(103.106.59.25:30303), IpV6 TCP Socket: None, Other Pairs: [("eth", "c68448fa7b0580"), ("secp256k1", "a102cfbeed345bc3a2a3a3e7354d74c38e9565c9bd4ba22cbc7d6f64c6f29f42759c")], .. }
2024-08-05T09:17:27.879711Z  INFO consensus::parlia: started listening to network block event
2024-08-05T09:17:27.879723Z  INFO consensus::parlia: started fork choice notifier
2024-08-05T09:17:27.879729Z  INFO consensus::parlia: started chain tracker notifier
2024-08-05T09:17:27.879777Z  INFO reth::cli: Pruner initialized prune_config=PruneConfig { block_interval: 5, segments: PruneModes { sender_recovery: Some(Full), transaction_lookup: None, receipts: None, account_history: Some(Distance(10064)), storage_history: Some(Distance(10064)), receipts_log_filter: ReceiptsLogPruneConfig({}) } }
2024-08-05T09:17:27.880364Z DEBUG consensus::engine: Pipeline sync progress is inconsistent first_stage_checkpoint=41053992 inconsistent_stage_id=Bodies inconsistent_stage_checkpoint=41053632
2024-08-05T09:17:27.880380Z  INFO reth::cli: Consensus engine initialized
2024-08-05T09:17:27.880414Z  INFO reth::cli: Engine API handler initialized
2024-08-05T09:17:27.880423Z DEBUG reth::cli: Reading JWT auth secret file path="/home/ubuntu/reth/bsc/jwt.hex"
2024-08-05T09:17:27.880425Z DEBUG consensus::parlia: Fetching new header block_hash=Number(41053633)
2024-08-05T09:17:27.880559Z DEBUG reth::cli: Using RPC module config http=Some(All) ws=Some(All)
2024-08-05T09:17:27.884027Z  INFO reth::cli: RPC auth server started url=127.0.0.1:8551
2024-08-05T09:17:27.884314Z  INFO reth::cli: RPC IPC server started path=/tmp/reth.ipc
2024-08-05T09:17:27.884323Z  INFO reth::cli: RPC HTTP server started url=127.0.0.1:8547
2024-08-05T09:17:27.884326Z  INFO reth::cli: RPC WS server started url=127.0.0.1:8548
2024-08-05T09:17:27.884383Z  INFO reth::cli: Starting consensus engine
2024-08-05T09:17:27.884475Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-08-05T09:17:27.884571Z DEBUG static_file: StaticFileProducer started targets=StaticFileTargets { headers: None, receipts: None, transactions: None, sidecars: Some(41053632..=41053632) }
2024-08-05T09:17:27.884581Z  INFO reth_node_events::node: Static File Producer started targets=StaticFileTargets { headers: None, receipts: None, transactions: None, sidecars: Some(41053632..=41053632) }
2024-08-05T09:17:27.884597Z DEBUG static_file: StaticFileProducer segment segment=Sidecars block_range=41053632..=41053632
2024-08-05T09:17:27.884638Z DEBUG static_file: Finished StaticFileProducer segment segment=Sidecars block_range=41053632..=41053632 elapsed=31.481µs
2024-08-05T09:17:27.884767Z DEBUG static_file: StaticFileProducer finished targets=StaticFileTargets { headers: None, receipts: None, transactions: None, sidecars: Some(41053632..=41053632) } elapsed=179.257µs
2024-08-05T09:17:27.884792Z  INFO reth_node_events::node: Static File Producer finished targets=StaticFileTargets { headers: None, receipts: None, transactions: None, sidecars: Some(41053632..=41053632) } elapsed=179.257µs
2024-08-05T09:17:27.884859Z DEBUG pruner: Pruner started tip_block_number=41053632
2024-08-05T09:17:27.884907Z DEBUG pruner: Segment pruning started segment=Transactions purpose=StaticFile to_block=41053632 prune_mode=Before(41053633)
2024-08-05T09:17:27.884938Z DEBUG pruner: Segment pruning finished segment=Transactions purpose=StaticFile to_block=41053632 prune_mode=Before(41053633) output.pruned=0
2024-08-05T09:17:27.884945Z DEBUG pruner: Nothing to prune for the segment segment=Headers purpose=StaticFile
2024-08-05T09:17:27.884953Z DEBUG pruner: Segment pruning started segment=Receipts purpose=StaticFile to_block=41053632 prune_mode=Before(41053633)
2024-08-05T09:17:27.884962Z DEBUG pruner: Segment pruning finished segment=Receipts purpose=StaticFile to_block=41053632 prune_mode=Before(41053633) output.pruned=0
2024-08-05T09:17:27.884984Z DEBUG storage::db::mdbx: Commit total_duration=16.081µs commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 0, audit: 0, write: 0, sync: 0, ending: 0, whole: 1, gc_cputime: 0, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0 } })) is_read_only=false
2024-08-05T09:17:27.885029Z DEBUG pruner: Pruner finished tip_block_number=41053632 elapsed=159.467µs deleted_entries=0 limiter=PruneLimiter { deleted_entries_limit: Some(PruneDeletedEntriesLimit { limit: 18446744073709551615, deleted: 0 }), time_limit: None } progress=Finished stats=[]
2024-08-05T09:17:27.885064Z  INFO reth_node_events::node: Preparing stage pipeline_stages=1/12 stage=Headers checkpoint=41053992 target=None
2024-08-05T09:17:27.885068Z  INFO sync::stages::headers: Target block already reached checkpoint=41053992 target=Hash(0xf28c6a3f3d06a8018c4e3138ca35d18493929f583e73111e5cafbb6cb71ef322)
2024-08-05T09:17:27.885085Z  INFO reth_node_events::node: Executing stage pipeline_stages=1/12 stage=Headers checkpoint=41053992 target=None
2024-08-05T09:17:27.885130Z  INFO reth_node_events::node: Finished stage pipeline_stages=1/12 stage=Headers checkpoint=41053992 target=None stage_progress=100.00%
2024-08-05T09:17:27.885140Z DEBUG storage::db::mdbx: Commit total_duration=13.59µs commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 0, audit: 0, write: 0, sync: 0, ending: 0, whole: 1, gc_cputime: 0, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0 } })) is_read_only=false
2024-08-05T09:17:27.885164Z  INFO reth_node_events::node: Preparing stage pipeline_stages=2/12 stage=Bodies checkpoint=41053632 target=41053992
2024-08-05T09:17:27.885173Z  INFO downloaders::bodies: Downloading bodies count=360 range=41053633..=41053992
2024-08-05T09:17:30.882074Z DEBUG consensus::parlia: Fetching new header block_hash=Number(41053633)
2024-08-05T09:17:30.923194Z  INFO reth::cli: Status connected_peers=0 freelist=22614074 stage=Bodies checkpoint=41053632 target=41053992
2024-08-05T09:17:30.973300Z DEBUG storage::db::mdbx: Commit total_duration=3.53µs commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 0, audit: 0, write: 0, sync: 0, ending: 0, whole: 1, gc_cputime: 0, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0 } })) is_read_only=true
2024-08-05T09:17:33.883303Z DEBUG consensus::parlia: Fetching new header block_hash=Number(41053633)
2024-08-05T09:17:36.883793Z DEBUG consensus::parlia: Fetching new header block_hash=Number(41053633)
2024-08-05T09:17:39.885456Z DEBUG consensus::parlia: Fetching new header block_hash=Number(41053633)
2024-08-05T09:17:42.887000Z DEBUG consensus::parlia: Fetching new header block_hash=Number(41053633)
2024-08-05T09:17:45.754963Z DEBUG storage::db::mdbx: Commit total_duration=3.38µs commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 0, audit: 0, write: 0, sync: 0, ending: 0, whole: 1, gc_cputime: 0, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0 } })) is_read_only=true
2024-08-05T09:17:45.887760Z DEBUG consensus::parlia: Fetching new header block_hash=Number(41053633)

Currently, I cannot connect to more nodes and cannot sync to the latest block. I feel that the data may have been corrupted. I have tried syncing through multiple nodes several times this month, and each time, an error occurs after completing all stages. I use PM2 to restart the node every time it crashes, but the program is still stuck at this point.

image
image
image

@greyireland
Copy link
Author

If you want more logs, you can contact me
TG:@greyireland

@sn00wman
Copy link

sn00wman commented Aug 6, 2024

I have encountered similar problem, if have a solution, please let me know.

@pythonberg1997
Copy link
Contributor

pythonberg1997 commented Aug 6, 2024

I am not using the snapshots above, I am using a synchronized full node. cmd

RUST_LOG=debug exec ./reth node --full --datadir /home/ubuntu/reth/bsc --datadir.static_files /home/ubuntu/data/rethdata/static_files --http --http.api all --http.addr 127.0.0.1 --http.port 8547 --ws --ws.api all --ws.addr 127.0.0.1 --ws.port 8548 --metrics 0.0.0.0:9001

debug logs

2024-08-05T09:17:27.782163Z  INFO reth::cli: Initialized tracing, debug log directory: /home/ubuntu/.cache/reth/logs/bsc
2024-08-05T09:17:27.785039Z  INFO reth::cli: Starting reth version="1.0.1 (c3361c97)"
2024-08-05T09:17:27.785653Z  INFO reth::cli: Opening database path="/home/ubuntu/reth/bsc/db"
2024-08-05T09:17:27.826185Z DEBUG reth_node_builder::launch::common: Raised file descriptor limit from=1048576 to=1048576
2024-08-05T09:17:27.830875Z  INFO reth::cli: Configuration loaded path="/home/ubuntu/reth/bsc/reth.toml"
2024-08-05T09:17:27.842510Z  INFO reth::cli: Verifying storage consistency.
2024-08-05T09:17:27.845891Z  INFO reth::cli: Database opened
2024-08-05T09:17:27.845903Z  INFO reth::cli: Starting metrics endpoint addr=0.0.0.0:9001
2024-08-05T09:17:27.845983Z DEBUG reth::cli: Initializing genesis chain=bsc genesis=0x0d21840abff46b96c84b2ac9e10e4f5cdaeb5693cb665db62a2f3b02d2d57b5b
2024-08-05T09:17:27.846034Z DEBUG reth_db_common::init: Genesis already written, skipping.
2024-08-05T09:17:27.846043Z  INFO reth::cli:
Pre-merge hard forks (block based):
- Frontier                         @0
- Homestead                        @0
- Tangerine                        @0
- SpuriousDragon                   @0
- Byzantium                        @0
- Constantinople                   @0
- Petersburg                       @0
- Istanbul                         @0
- MuirGlacier                      @0
- Ramanujan                        @0
- Niels                            @0
- MirrorSync                       @5184000
- Bruno                            @13082000
- Euler                            @18907621
- Nano                             @21962149
- Moran                            @22107423
- Gibbs                            @23846001
- Planck                           @27281024
- Luban                            @29020050
- Plato                            @30720096
- Berlin                           @31302048
- London                           @31302048
- Hertz                            @31302048
- HertzFix                         @34140700
Post-merge hard forks (timestamp based):
- Shanghai                         @1705996800
- Kepler                           @1705996800
- Feynman                          @1713419340
- FeynmanFix                       @1713419340
- Cancun                           @1718863500
- Haber                            @1718863500
2024-08-05T09:17:27.846068Z DEBUG reth::cli: Spawning stages metrics listener task
2024-08-05T09:17:27.846118Z DEBUG reth::cli: creating components
2024-08-05T09:17:27.846184Z DEBUG txpool::blob: Removed blob store directory blob_dir="/home/ubuntu/reth/bsc/blobstore"
2024-08-05T09:17:27.846192Z DEBUG txpool::blob: Creating blob store blob_dir="/home/ubuntu/reth/bsc/blobstore"
2024-08-05T09:17:27.846367Z  INFO reth::cli: Transaction pool initialized
2024-08-05T09:17:27.846386Z DEBUG reth::cli: Spawned txpool maintenance task
2024-08-05T09:17:27.850945Z  INFO net::peers: Loading saved peers file=/home/ubuntu/reth/bsc/known-peers.json
2024-08-05T09:17:27.851861Z DEBUG discv4: pinging boot node record=NodeRecord { address: 52.69.102.73, tcp_port: 30311, udp_port: 30311, id: 0x433c8bfdf53a3e2268ccb1b829e47f629793291cbddf0c76ae626da802f90532251fc558e2e0d10d6725e759088439bf1cd4714716b03a259a35d4b2e4acfa7f }
2024-08-05T09:17:27.851954Z DEBUG discv4: pinging boot node record=NodeRecord { address: 18.203.152.54, tcp_port: 30311, udp_port: 30311, id: 0xfac42fb0ba082b7d1eebded216db42161163d42e4f52c9e47716946d64468a62da4ba0b1cac0df5e8bf1e5284861d757339751c33d51dfef318be5168803d0b5 }
2024-08-05T09:17:27.851992Z DEBUG discv4: pinging boot node record=NodeRecord { address: 34.250.32.100, tcp_port: 30311, udp_port: 30311, id: 0x3063d1c9e1b824cfbb7c7b6abafa34faec6bb4e7e06941d218d760acdd7963b274278c5c3e63914bd6d1b58504c59ec5522c56f883baceb8538674b92da48a96 }
2024-08-05T09:17:27.852026Z DEBUG discv4: pinging boot node record=NodeRecord { address: 35.73.84.120, tcp_port: 30311, udp_port: 30311, id: 0x571bee8fb902a625942f10a770ccf727ae2ba1bab2a2b64e121594a99c9437317f6166a395670a00b7d93647eacafe598b6bbcef15b40b6d1a10243865a3e80f }
2024-08-05T09:17:27.852065Z DEBUG discv4: pinging boot node record=NodeRecord { address: 34.204.214.24, tcp_port: 30311, udp_port: 30311, id: 0xad78c64a4ade83692488aa42e4c94084516e555d3f340d9802c2bf106a3df8868bc46eae083d2de4018f40e8d9a9952c32a0943cd68855a9bc9fd07aac982a6d }
2024-08-05T09:17:27.852100Z DEBUG discv4: pinging boot node record=NodeRecord { address: 107.20.191.97, tcp_port: 30311, udp_port: 30311, id: 0x5db798deb67df75d073f8e2953dad283148133acb520625ea804c9c4ad09a35f13592a762d8f89056248f3889f6dcc33490c145774ea4ff2966982294909b37a }
2024-08-05T09:17:27.852872Z DEBUG reth::cli: configured blockchain tree
2024-08-05T09:17:27.852882Z DEBUG reth::cli: calling on_component_initialized hook
2024-08-05T09:17:27.853219Z  INFO reth::cli: StaticFileProducer initialized
2024-08-05T09:17:27.853505Z DEBUG discv4: pinging boot node record=NodeRecord { address: 52.69.102.73, tcp_port: 30311, udp_port: 30311, id: 0x433c8bfdf53a3e2268ccb1b829e47f629793291cbddf0c76ae626da802f90532251fc558e2e0d10d6725e759088439bf1cd4714716b03a259a35d4b2e4acfa7f }
2024-08-05T09:17:27.853546Z DEBUG discv4: pinging boot node record=NodeRecord { address: 18.203.152.54, tcp_port: 30311, udp_port: 30311, id: 0xfac42fb0ba082b7d1eebded216db42161163d42e4f52c9e47716946d64468a62da4ba0b1cac0df5e8bf1e5284861d757339751c33d51dfef318be5168803d0b5 }
2024-08-05T09:17:27.853552Z DEBUG discv4: pinging boot node record=NodeRecord { address: 34.250.32.100, tcp_port: 30311, udp_port: 30311, id: 0x3063d1c9e1b824cfbb7c7b6abafa34faec6bb4e7e06941d218d760acdd7963b274278c5c3e63914bd6d1b58504c59ec5522c56f883baceb8538674b92da48a96 }
2024-08-05T09:17:27.853555Z DEBUG discv4: pinging boot node record=NodeRecord { address: 35.73.84.120, tcp_port: 30311, udp_port: 30311, id: 0x571bee8fb902a625942f10a770ccf727ae2ba1bab2a2b64e121594a99c9437317f6166a395670a00b7d93647eacafe598b6bbcef15b40b6d1a10243865a3e80f }
2024-08-05T09:17:27.853559Z DEBUG discv4: pinging boot node record=NodeRecord { address: 34.204.214.24, tcp_port: 30311, udp_port: 30311, id: 0xad78c64a4ade83692488aa42e4c94084516e555d3f340d9802c2bf106a3df8868bc46eae083d2de4018f40e8d9a9952c32a0943cd68855a9bc9fd07aac982a6d }
2024-08-05T09:17:27.853564Z DEBUG discv4: pinging boot node record=NodeRecord { address: 107.20.191.97, tcp_port: 30311, udp_port: 30311, id: 0x5db798deb67df75d073f8e2953dad283148133acb520625ea804c9c4ad09a35f13592a762d8f89056248f3889f6dcc33490c145774ea4ff2966982294909b37a }
2024-08-05T09:17:27.853654Z DEBUG reqwest::connect: starting new connection: http://ipinfo.io/
2024-08-05T09:17:27.853732Z DEBUG reqwest::connect: starting new connection: http://icanhazip.com/
2024-08-05T09:17:27.853800Z DEBUG reqwest::connect: starting new connection: http://ifconfig.me/
2024-08-05T09:17:27.853890Z DEBUG hyper_util::client::legacy::connect::dns: resolving host="ipinfo.io"
2024-08-05T09:17:27.854017Z DEBUG hyper_util::client::legacy::connect::dns: resolving host="icanhazip.com"
2024-08-05T09:17:27.854061Z DEBUG hyper_util::client::legacy::connect::dns: resolving host="ifconfig.me"
2024-08-05T09:17:27.854891Z DEBUG hyper_util::client::legacy::connect::http: connecting to 104.16.185.241:80
2024-08-05T09:17:27.855210Z DEBUG hyper_util::client::legacy::connect::http: connecting to 34.117.59.81:80
2024-08-05T09:17:27.855247Z DEBUG hyper_util::client::legacy::connect::http: connecting to 34.160.111.145:80
2024-08-05T09:17:27.855819Z DEBUG hyper_util::client::legacy::connect::http: connected to 104.16.185.241:80
2024-08-05T09:17:27.856363Z DEBUG hyper_util::client::legacy::connect::http: connected to 34.117.59.81:80
2024-08-05T09:17:27.856624Z DEBUG hyper_util::client::legacy::connect::http: connected to 34.160.111.145:80
2024-08-05T09:17:27.864957Z DEBUG hyper_util::client::legacy::pool: pooling idle connection for ("http", icanhazip.com)
2024-08-05T09:17:27.865007Z DEBUG discv4: Updating external ip external_ip=103.106.59.25
2024-08-05T09:17:27.865178Z DEBUG discv4: Updated local ENR enr=Enr { id: Some("v4"), seq: 2, NodeId: 0x9576166c8569842ba094ace22087fee14ad691efed63cd3bb4ecb1e87c653e71, signature: "eb955bb1d1c8d1b16d4055d00a3c510b621bc6f5390cf0304b29c106d42057b63becb65fddb526d1e0cb1591e31b6f6d1207405c0f726cdf3cb60779b18a8ec0", IpV4 UDP Socket: Some(103.106.59.25:30303), IpV6 UDP Socket: None, IpV4 TCP Socket: Some(103.106.59.25:30303), IpV6 TCP Socket: None, Other Pairs: [("eth", "c68448fa7b0580"), ("secp256k1", "a102cfbeed345bc3a2a3a3e7354d74c38e9565c9bd4ba22cbc7d6f64c6f29f42759c")], .. }
2024-08-05T09:17:27.879711Z  INFO consensus::parlia: started listening to network block event
2024-08-05T09:17:27.879723Z  INFO consensus::parlia: started fork choice notifier
2024-08-05T09:17:27.879729Z  INFO consensus::parlia: started chain tracker notifier
2024-08-05T09:17:27.879777Z  INFO reth::cli: Pruner initialized prune_config=PruneConfig { block_interval: 5, segments: PruneModes { sender_recovery: Some(Full), transaction_lookup: None, receipts: None, account_history: Some(Distance(10064)), storage_history: Some(Distance(10064)), receipts_log_filter: ReceiptsLogPruneConfig({}) } }
2024-08-05T09:17:27.880364Z DEBUG consensus::engine: Pipeline sync progress is inconsistent first_stage_checkpoint=41053992 inconsistent_stage_id=Bodies inconsistent_stage_checkpoint=41053632
2024-08-05T09:17:27.880380Z  INFO reth::cli: Consensus engine initialized
2024-08-05T09:17:27.880414Z  INFO reth::cli: Engine API handler initialized
2024-08-05T09:17:27.880423Z DEBUG reth::cli: Reading JWT auth secret file path="/home/ubuntu/reth/bsc/jwt.hex"
2024-08-05T09:17:27.880425Z DEBUG consensus::parlia: Fetching new header block_hash=Number(41053633)
2024-08-05T09:17:27.880559Z DEBUG reth::cli: Using RPC module config http=Some(All) ws=Some(All)
2024-08-05T09:17:27.884027Z  INFO reth::cli: RPC auth server started url=127.0.0.1:8551
2024-08-05T09:17:27.884314Z  INFO reth::cli: RPC IPC server started path=/tmp/reth.ipc
2024-08-05T09:17:27.884323Z  INFO reth::cli: RPC HTTP server started url=127.0.0.1:8547
2024-08-05T09:17:27.884326Z  INFO reth::cli: RPC WS server started url=127.0.0.1:8548
2024-08-05T09:17:27.884383Z  INFO reth::cli: Starting consensus engine
2024-08-05T09:17:27.884475Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-08-05T09:17:27.884571Z DEBUG static_file: StaticFileProducer started targets=StaticFileTargets { headers: None, receipts: None, transactions: None, sidecars: Some(41053632..=41053632) }
2024-08-05T09:17:27.884581Z  INFO reth_node_events::node: Static File Producer started targets=StaticFileTargets { headers: None, receipts: None, transactions: None, sidecars: Some(41053632..=41053632) }
2024-08-05T09:17:27.884597Z DEBUG static_file: StaticFileProducer segment segment=Sidecars block_range=41053632..=41053632
2024-08-05T09:17:27.884638Z DEBUG static_file: Finished StaticFileProducer segment segment=Sidecars block_range=41053632..=41053632 elapsed=31.481µs
2024-08-05T09:17:27.884767Z DEBUG static_file: StaticFileProducer finished targets=StaticFileTargets { headers: None, receipts: None, transactions: None, sidecars: Some(41053632..=41053632) } elapsed=179.257µs
2024-08-05T09:17:27.884792Z  INFO reth_node_events::node: Static File Producer finished targets=StaticFileTargets { headers: None, receipts: None, transactions: None, sidecars: Some(41053632..=41053632) } elapsed=179.257µs
2024-08-05T09:17:27.884859Z DEBUG pruner: Pruner started tip_block_number=41053632
2024-08-05T09:17:27.884907Z DEBUG pruner: Segment pruning started segment=Transactions purpose=StaticFile to_block=41053632 prune_mode=Before(41053633)
2024-08-05T09:17:27.884938Z DEBUG pruner: Segment pruning finished segment=Transactions purpose=StaticFile to_block=41053632 prune_mode=Before(41053633) output.pruned=0
2024-08-05T09:17:27.884945Z DEBUG pruner: Nothing to prune for the segment segment=Headers purpose=StaticFile
2024-08-05T09:17:27.884953Z DEBUG pruner: Segment pruning started segment=Receipts purpose=StaticFile to_block=41053632 prune_mode=Before(41053633)
2024-08-05T09:17:27.884962Z DEBUG pruner: Segment pruning finished segment=Receipts purpose=StaticFile to_block=41053632 prune_mode=Before(41053633) output.pruned=0
2024-08-05T09:17:27.884984Z DEBUG storage::db::mdbx: Commit total_duration=16.081µs commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 0, audit: 0, write: 0, sync: 0, ending: 0, whole: 1, gc_cputime: 0, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0 } })) is_read_only=false
2024-08-05T09:17:27.885029Z DEBUG pruner: Pruner finished tip_block_number=41053632 elapsed=159.467µs deleted_entries=0 limiter=PruneLimiter { deleted_entries_limit: Some(PruneDeletedEntriesLimit { limit: 18446744073709551615, deleted: 0 }), time_limit: None } progress=Finished stats=[]
2024-08-05T09:17:27.885064Z  INFO reth_node_events::node: Preparing stage pipeline_stages=1/12 stage=Headers checkpoint=41053992 target=None
2024-08-05T09:17:27.885068Z  INFO sync::stages::headers: Target block already reached checkpoint=41053992 target=Hash(0xf28c6a3f3d06a8018c4e3138ca35d18493929f583e73111e5cafbb6cb71ef322)
2024-08-05T09:17:27.885085Z  INFO reth_node_events::node: Executing stage pipeline_stages=1/12 stage=Headers checkpoint=41053992 target=None
2024-08-05T09:17:27.885130Z  INFO reth_node_events::node: Finished stage pipeline_stages=1/12 stage=Headers checkpoint=41053992 target=None stage_progress=100.00%
2024-08-05T09:17:27.885140Z DEBUG storage::db::mdbx: Commit total_duration=13.59µs commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 0, audit: 0, write: 0, sync: 0, ending: 0, whole: 1, gc_cputime: 0, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0 } })) is_read_only=false
2024-08-05T09:17:27.885164Z  INFO reth_node_events::node: Preparing stage pipeline_stages=2/12 stage=Bodies checkpoint=41053632 target=41053992
2024-08-05T09:17:27.885173Z  INFO downloaders::bodies: Downloading bodies count=360 range=41053633..=41053992
2024-08-05T09:17:30.882074Z DEBUG consensus::parlia: Fetching new header block_hash=Number(41053633)
2024-08-05T09:17:30.923194Z  INFO reth::cli: Status connected_peers=0 freelist=22614074 stage=Bodies checkpoint=41053632 target=41053992
2024-08-05T09:17:30.973300Z DEBUG storage::db::mdbx: Commit total_duration=3.53µs commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 0, audit: 0, write: 0, sync: 0, ending: 0, whole: 1, gc_cputime: 0, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0 } })) is_read_only=true
2024-08-05T09:17:33.883303Z DEBUG consensus::parlia: Fetching new header block_hash=Number(41053633)
2024-08-05T09:17:36.883793Z DEBUG consensus::parlia: Fetching new header block_hash=Number(41053633)
2024-08-05T09:17:39.885456Z DEBUG consensus::parlia: Fetching new header block_hash=Number(41053633)
2024-08-05T09:17:42.887000Z DEBUG consensus::parlia: Fetching new header block_hash=Number(41053633)
2024-08-05T09:17:45.754963Z DEBUG storage::db::mdbx: Commit total_duration=3.38µs commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 0, audit: 0, write: 0, sync: 0, ending: 0, whole: 1, gc_cputime: 0, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0 } })) is_read_only=true
2024-08-05T09:17:45.887760Z DEBUG consensus::parlia: Fetching new header block_hash=Number(41053633)

Currently, I cannot connect to more nodes and cannot sync to the latest block. I feel that the data may have been corrupted. I have tried syncing through multiple nodes several times this month, and each time, an error occurs after completing all stages. I use PM2 to restart the node every time it crashes, but the program is still stuck at this point.

image image image

Let's focus on the peer issue first. Could you please set RUST_LOG=trace and restart the node(use the develop branch to build) and show the logs again? @greyireland

@greyireland
Copy link
Author

If anyone has successfully run this bsc-reth, I hope you can give some good running suggestions. I have tried syncing many times with various methods, but none of them worked. I have given up.

@shouc
Copy link

shouc commented Aug 13, 2024

same problem here...

@wesraph
Copy link

wesraph commented Aug 13, 2024

same issue

@pythonberg1997
Copy link
Contributor

Can you provide the debug log when this error happened please? @wesraph @shouc

@unclezoro
Copy link
Collaborator

hi, guys. Please join the TG group https://t.me/+M9D9Eo7sMj4wNDI1, so that we can provide support in time

@pythonberg1997
Copy link
Contributor

This issue was caused by database corruption due to an unexpected event. I will close this issue for now. If anyone encounters the same problem(Error: database integrity error occurred over and over again), please reopen it or join the TG group to provide feedback.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants