Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

nodeos 3.1.0 and 4.0.3 cannot sync EOS from genesis with ship enabled #1354

Closed
matthewdarwin opened this issue Jun 30, 2023 · 2 comments · Fixed by #1379, #1380 or #1382
Closed

nodeos 3.1.0 and 4.0.3 cannot sync EOS from genesis with ship enabled #1354

matthewdarwin opened this issue Jun 30, 2023 · 2 comments · Fixed by #1379, #1380 or #1382
Assignees
@heifner
Labels
actionable bug Something isn't working 👍 lgtm OCI Work exclusive to OCI team
Milestone
Leap 3.2.4

Comments

@matthewdarwin
Copy link

matthewdarwin commented Jun 30, 2023 •
edited
Loading

On the EOS network, nodeos 3.1.0 and 4.0.3 cannot sync from genesis with ship enabled. Always crashes at block 1225984. Here are logs from version 4.0.3:

Jun 30 12:30:00 eos-ship46 nodeos[10771]: info  2023-06-30T12:30:00.682 net-9     net_plugin.cpp:1622           request_next_chunk   ] ignoring request, head is 1225983 last req = 1228148, sync_next_expected_num: 1225984, s
ync_known_lib_num: 317885317, sync_req_span: 2500, source connection 2
Jun 30 12:30:01 eos-ship46 nodeos[10771]: info  2023-06-30T12:30:01.051 nodeos    platform_timer_accurac:62     compute_and_print_ti ] Checktime timer accuracy: min:0us max:4232us mean:2us stddev:66us
Jun 30 12:30:01 eos-ship46 nodeos[10771]: error 2023-06-30T12:30:01.052 nodeos    state_history_plugin.c:286    on_accepted_block    ] fc::exception: 3110000 plugin_exception: Plugin exception
Jun 30 12:30:01 eos-ship46 nodeos[10771]: missing trace for transaction abb5d1965cf2d7237a7f15fb0884ca3b4409aad53e2a99e00b072bf9d60aa1a1
Jun 30 12:30:01 eos-ship46 nodeos[10771]:     {"id":"abb5d1965cf2d7237a7f15fb0884ca3b4409aad53e2a99e00b072bf9d60aa1a1"}
Jun 30 12:30:01 eos-ship46 nodeos[10771]:     nodeos  trace_converter.cpp:30 pack
Jun 30 12:30:01 eos-ship46 nodeos[10771]: warn  2023-06-30T12:30:01.052 nodeos    controller.cpp:392            emit                 ] controller_emit_signal_exception: 3140002 state_history_write_exception: State history write error
Jun 30 12:30:01 eos-ship46 nodeos[10771]: State history encountered an Error which it cannot recover from.  Please resolve the error and relaunch the process
Jun 30 12:30:01 eos-ship46 nodeos[10771]:     {}
Jun 30 12:30:01 eos-ship46 nodeos[10771]:     nodeos  state_history_plugin.cpp:294 on_accepted_block
Jun 30 12:30:01 eos-ship46 nodeos[10771]: error 2023-06-30T12:30:01.052 nodeos    controller.cpp:2171           apply_block          ] e.to_detail_string(): 3140002 state_history_write_exception: State history write error
Jun 30 12:30:01 eos-ship46 nodeos[10771]: State history encountered an Error which it cannot recover from.  Please resolve the error and relaunch the process
Jun 30 12:30:01 eos-ship46 nodeos[10771]:     {}
Jun 30 12:30:01 eos-ship46 nodeos[10771]:     nodeos  state_history_plugin.cpp:294 on_accepted_block
Jun 30 12:30:01 eos-ship46 nodeos[10771]:
Jun 30 12:30:01 eos-ship46 nodeos[10771]: warn  2023-06-30T12:30:01.052 nodeos    controller.cpp:2273           push_block           ] 3140002 state_history_write_exception: State history write error
Jun 30 12:30:01 eos-ship46 nodeos[10771]: State history encountered an Error which it cannot recover from.  Please resolve the error and relaunch the process
Jun 30 12:30:01 eos-ship46 nodeos[10771]:     {}
Jun 30 12:30:01 eos-ship46 nodeos[10771]:     nodeos  state_history_plugin.cpp:294 on_accepted_block
Jun 30 12:30:01 eos-ship46 nodeos[10771]:     {}
Jun 30 12:30:01 eos-ship46 nodeos[10771]:     nodeos  controller.cpp:2179 apply_block
Jun 30 12:30:01 eos-ship46 nodeos[10771]: error 2023-06-30T12:30:01.052 nodeos    producer_plugin.cpp:608       operator()           ] Exception on block 1225984: 3140002 state_history_write_exception: State history write error
Jun 30 12:30:01 eos-ship46 nodeos[10771]: State history encountered an Error which it cannot recover from.  Please resolve the error and relaunch the process
Jun 30 12:30:01 eos-ship46 nodeos[10771]:     {}
Jun 30 12:30:01 eos-ship46 nodeos[10771]:     nodeos  state_history_plugin.cpp:294 on_accepted_block
Jun 30 12:30:01 eos-ship46 nodeos[10771]:     {}
Jun 30 12:30:01 eos-ship46 nodeos[10771]:     nodeos  controller.cpp:2179 apply_block
Jun 30 12:30:01 eos-ship46 nodeos[10771]: rethrow
Jun 30 12:30:01 eos-ship46 nodeos[10771]:     {}
Jun 30 12:30:01 eos-ship46 nodeos[10771]:     nodeos  controller.cpp:2273 push_block
Jun 30 12:30:01 eos-ship46 nodeos[10771]: error 2023-06-30T12:30:01.052 nodeos    net_plugin.cpp:3325           process_signed_block ] bad block exception connection 2: #1225984 ff1f0e4a2372e441...: State history write error (3140002)
Jun 30 12:30:01 eos-ship46 nodeos[10771]: State history encountered an Error which it cannot recover from.  Please resolve the error and relaunch the process
Jun 30 12:30:01 eos-ship46 nodeos[10771]: rethrow
@enf-ci-bot enf-ci-bot moved this to Todo in Team Backlog Jun 30, 2023
@matthewdarwin matthewdarwin mentioned this issue Jun 30, 2023
Closed
@eosusa
Copy link

eosusa commented Jun 30, 2023 •
edited
Loading

Also tested disabling SHiP to sync node past that block, which it successfully did, then attempted a --replay-blockchain shortly after (with SHiP) re-enabled, and get the same error about missing trace for that transaction:

info  2023-06-30T12:44:51.750 nodeos    controller.cpp:495            replay               ] 1225000 of 1428536
info  2023-06-30T12:44:51.826 nodeos    controller.cpp:495            replay               ] 1225500 of 1428536
error 2023-06-30T12:44:51.901 nodeos    state_history_plugin.c:364    on_accepted_block    ] fc::exception: 3110000 plugin_exception: Plugin exception
missing trace for transaction abb5d1965cf2d7237a7f15fb0884ca3b4409aad53e2a99e00b072bf9d60aa1a1
    {"id":"abb5d1965cf2d7237a7f15fb0884ca3b4409aad53e2a99e00b072bf9d60aa1a1"}
    nodeos  trace_converter.cpp:30 pack

warn  2023-06-30T12:44:51.901 nodeos    controller.cpp:376            emit                 ] controller_emit_signal_exception: 3140002 state_history_write_exception: State history write error
State history encountered an Error which it cannot recover from.  Please resolve the error and relaunch the process
    {}
    nodeos  state_history_plugin.cpp:372 on_accepted_block

error 2023-06-30T12:44:51.901 nodeos    controller.cpp:2048           apply_block          ] e.to_detail_string(): 3140002 state_history_write_exception: State history write error
State history encountered an Error which it cannot recover from.  Please resolve the error and relaunch the process
    {}
    nodeos  state_history_plugin.cpp:372 on_accepted_block

warn  2023-06-30T12:44:51.901 nodeos    controller.cpp:2185           replay_push_block    ] 3140002 state_history_write_exception: State history write error
State history encountered an Error which it cannot recover from.  Please resolve the error and relaunch the process
    {}
    nodeos  state_history_plugin.cpp:372 on_accepted_block

    {}
    nodeos  controller.cpp:2056 apply_block

error 2023-06-30T12:44:51.975 nodeos    main.cpp:157                  main                 ] 3140002 state_history_write_exception: State history write error
State history encountered an Error which it cannot recover from.  Please resolve the error and relaunch the process
    {}
    nodeos  state_history_plugin.cpp:372 on_accepted_block

    {}
    nodeos  controller.cpp:2056 apply_block
rethrow
    {}
    nodeos  controller.cpp:2185 replay_push_block

    {}
    nodeos  chain_plugin.cpp:1290 plugin_startup

@spoonincode spoonincode added bug Something isn't working actionable and removed triage labels Jun 30, 2023
@arhag arhag added the 👍 lgtm label Jul 4, 2023
@heifner heifner added the OCI Work exclusive to OCI team label Jul 4, 2023
@heifner heifner self-assigned this Jul 4, 2023
@heifner
Copy link
Member

heifner commented Jul 4, 2023

In 2.x https://github.com/EOSIO/eos/blob/v2.0.13/libraries/chain/controller.cpp#L1259C7-L1259C88 the scheduled trx id is passed transaction_context and used for the transaction trace. In 2.1 & 3.1 the id is no longer passed to transaction_context for the transaction trace. This is a problem for scheduled transactions created before builtin_protocol_feature_t::replace_deferred as the transaction id of the schedule transaction does not match the id of the packed_transaction now passed to transaction_context.

Need to pass transaction id to transaction_context like what was done in 2.x.

@heifner heifner added this to the Leap 3.2.4 milestone Jul 6, 2023
heifner added a commit that referenced this issue Jul 7, 2023
@heifner
GH-1354 Add test case for deferred trx id before/after protocol featu…
97aa4b8 
…re replace_deferred
heifner added a commit that referenced this issue Jul 7, 2023
@heifner
GH-1354 Provide transaction_context trx id for trace since it can dif…
71ecec4 
…fer from provided packed_transaction for deferred trxs before replace_deferred protocol feature is activated.
@heifner heifner mentioned this issue Jul 7, 2023
Merged
@heifner heifner linked a pull request Jul 7, 2023 that will close this issue
[3.2] Emit correct trace id for deferred trx before replace_deferred protocol feature activation #1379
Merged
@heifner heifner moved this from Todo to In Progress in Team Backlog Jul 7, 2023
@heifner heifner moved this from In Progress to Awaiting Review in Team Backlog Jul 7, 2023
heifner added a commit that referenced this issue Jul 7, 2023
@heifner
Merge pull request #1379 from AntelopeIO/GH-1354-deferred-id-3.2
4c1226e 
[3.2] Emit correct trace id for deferred trx before replace_deferred protocol feature activation
heifner added a commit that referenced this issue Jul 7, 2023
@heifner
Merge remote-tracking branch 'origin/release/3.2' into GH-1354-deferr…
399ca84 
…ed-id-4.0
@heifner heifner mentioned this issue Jul 7, 2023
Merged
heifner added a commit that referenced this issue Jul 7, 2023
@heifner
Merge pull request #1380 from AntelopeIO/GH-1354-deferred-id-4.0
923904f 
[3.2 -> 4.0] Emit correct trace id for deferred trx before replace_deferred protocol feature activation
@github-project-automation github-project-automation bot moved this from Awaiting Review to Done in Team Backlog Jul 7, 2023
@heifner heifner mentioned this issue Jul 7, 2023
Merged
heifner added a commit that referenced this issue Jul 7, 2023
@heifner
Merge pull request #1381 from AntelopeIO/GH-1354-deferred-id-4.0
9cada46 
[3.2 -> 4.0] Emit correct trace id for deferred trx before replace_deferred protocol feature activation
heifner added a commit that referenced this issue Jul 7, 2023
@heifner
Merge remote-tracking branch 'origin/release/4.0' into GH-1354-deferr…
46ba947 
…ed-id-main
heifner added a commit that referenced this issue Jul 7, 2023
@heifner
Merge remote-tracking branch 'origin/release/4.0' into GH-1354-deferr…
0c18762 
…ed-id-m
@heifner heifner mentioned this issue Jul 7, 2023
Merged
heifner added a commit that referenced this issue Jul 7, 2023
@heifner
Merge pull request #1382 from AntelopeIO/GH-1354-deferred-id-m
1b79caa 
[4.0 -> main] Emit correct trace id for deferred trx before replace_deferred protocol feature activation
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@heifner heifner

Labels
actionable bug Something isn't working 👍 lgtm OCI Work exclusive to OCI team
Projects
Team Backlog
Archived in project
Milestone
Leap 3.2.4
6 participants
@heifner @arhag @matthewdarwin @spoonincode @eosusa @enf-ci-bot