Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

3.2 state_history_plugin crashes under heavy load #592

Closed
heifner opened this issue Dec 22, 2022 · 1 comment · Fixed by #638 or #628
Closed

3.2 state_history_plugin crashes under heavy load #592

heifner opened this issue Dec 22, 2022 · 1 comment · Fixed by #638 or #628
Assignees
@heifner
Labels
actionable bug Something isn't working 👍 lgtm OCI Work exclusive to OCI team

Comments

@heifner
Copy link
Member

heifner commented Dec 22, 2022

[Information reported by community]

3.1 does not have this issue. Observed with both v3.2.0-rc1 and v3.2.0.

Reproducible by indexing with Hyperion with high scale values.

image

Double these values to speed up the indexing and will reproduce the state_history_plugin crash.

image

OC enabled:
##############
info  2022-12-13T16:24:31.615 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T16:24:31.620 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T16:24:31.622 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T16:24:31.625 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T16:24:31.627 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T16:24:31.628 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T16:24:31.629 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T16:24:31.632 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T16:24:31.632 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T16:24:31.632 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T16:24:31.633 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T16:24:31.634 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T16:24:31.636 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T16:24:31.636 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T16:24:31.638 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T16:24:31.643 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T16:24:31.643 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T16:24:31.643 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T16:24:31.643 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
double free or corruption (out)
ERROR: EOS VM OC compiler monitor exiting with active sessions


info  2022-12-13T17:11:18.060 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T17:11:18.062 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T17:11:18.068 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T17:11:18.068 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T17:11:18.068 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T17:11:18.069 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T17:11:18.069 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
warn  2022-12-13T17:11:18.070 nodeos    block_log.cpp:699             read_block_by_num    ] 10 assert_exception: Assert Exception
Provided index out of range for variant.
    {}
    nodeos  static_variant.hpp:19 from_index
Error unpacking field trx
    {"field":"trx"}
    nodeos  raw.hpp:363 operator()
error unpacking eosio::chain::transaction_receipt
    {"type":"eosio::chain::transaction_receipt"}
    nodeos  raw.hpp:668 unpack
Error unpacking field transactions
    {"field":"transactions"}
    nodeos  raw.hpp:363 operator()
error unpacking eosio::chain::signed_block
    {"type":"eosio::chain::signed_block"}
    nodeos  raw.hpp:668 unpack

warn  2022-12-13T17:11:18.070 nodeos    block_log.cpp:699             read_block_by_num    ] 10 assert_exception: Assert Exception
Provided index out of range for variant.
    {}
    nodeos  static_variant.hpp:19 from_index
Error unpacking field _storage
    {"field":"_storage"}
    nodeos  raw.hpp:363 operator()
error unpacking fc::crypto::signature
    {"type":"fc::crypto::signature"}
    nodeos  raw.hpp:668 unpack
Error unpacking field signatures
    {"field":"signatures"}
    nodeos  raw.hpp:363 operator()
error unpacking eosio::chain::packed_transaction
    {"type":"eosio::chain::packed_transaction"}
    nodeos  raw.hpp:668 unpack
Error unpacking field trx
    {"field":"trx"}
    nodeos  raw.hpp:363 operator()
error unpacking eosio::chain::transaction_receipt
    {"type":"eosio::chain::transaction_receipt"}
    nodeos  raw.hpp:668 unpack
Error unpacking field transactions
    {"field":"transactions"}
    nodeos  raw.hpp:363 operator()
error unpacking eosio::chain::signed_block
    {"type":"eosio::chain::signed_block"}
    nodeos  raw.hpp:668 unpack

info  2022-12-13T17:11:18.070 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
warn  2022-12-13T17:11:18.070 nodeos    block_log.cpp:699             read_block_by_num    ] 10 assert_exception: Assert Exception
(b & ~1) == 0:
    {}
    nodeos  raw.hpp:334 unpack
optional<eosio::chain::legacy::producer_schedule_type>
    {"type":"eosio::chain::legacy::producer_schedule_type"}
    nodeos  raw.hpp:282 unpack
Error unpacking field new_producers
    {"field":"new_producers"}
    nodeos  raw.hpp:363 operator()
error unpacking eosio::chain::signed_block
    {"type":"eosio::chain::signed_block"}
    nodeos  raw.hpp:668 unpack

warn  2022-12-13T17:11:18.070 nodeos    block_log.cpp:699             read_block_by_num    ] 10 assert_exception: Assert Exception
Provided index out of range for variant.
    {}
    nodeos  static_variant.hpp:19 from_index
Error unpacking field trx
    {"field":"trx"}
    nodeos  raw.hpp:363 operator()
error unpacking eosio::chain::transaction_receipt
    {"type":"eosio::chain::transaction_receipt"}
    nodeos  raw.hpp:668 unpack
Error unpacking field transactions
    {"field":"transactions"}
    nodeos  raw.hpp:363 operator()
error unpacking eosio::chain::signed_block
    {"type":"eosio::chain::signed_block"}
    nodeos  raw.hpp:668 unpack


OC disabled
#############
info  2022-12-13T17:11:18.085 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T17:11:18.085 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T17:11:18.085 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T17:11:18.085 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T17:11:18.087 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T17:11:18.092 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T17:11:18.092 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T17:11:18.092 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T17:11:18.092 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T17:11:18.094 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T17:11:18.095 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T17:11:18.095 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T17:11:18.106 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T17:11:18.106 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T17:11:18.108 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T17:11:18.116 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T17:11:18.119 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T17:11:18.123 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T17:11:18.123 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T17:11:18.124 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T17:11:18.135 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T17:11:18.142 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T17:11:18.143 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
info  2022-12-13T17:11:18.144 nodeos    state_history_plugin.c:266    operator()           ] received get_blocks_ack_request_v0 = {"num_messages":1}
double free or corruption (out)
@enf-ci-bot enf-ci-bot moved this to Todo in Team Backlog Dec 22, 2022
@heifner heifner added bug Something isn't working actionable and removed triage labels Dec 22, 2022
huangminghuang added a commit that referenced this issue Dec 26, 2022
@huangminghuang
fix issue #592
83999b0
huangminghuang added a commit that referenced this issue Dec 26, 2022
@huangminghuang
fix issue #592
9329fa3
huangminghuang added a commit that referenced this issue Dec 28, 2022
@huangminghuang
fix issue #592
8471e40
@huangminghuang huangminghuang mentioned this issue Dec 28, 2022
Closed
huangminghuang added a commit that referenced this issue Dec 28, 2022
@huangminghuang
fix issue #592
9fba5f1
@huangminghuang huangminghuang mentioned this issue Dec 28, 2022
Closed
@stephenpdeos stephenpdeos mentioned this issue Jan 5, 2023
Closed
@heifner heifner self-assigned this Jan 5, 2023
@heifner heifner moved this from Todo to In Progress in Team Backlog Jan 5, 2023
@heifner heifner added the OCI Work exclusive to OCI team label Jan 5, 2023
heifner added a commit that referenced this issue Jan 7, 2023
@heifner
GH-592 Back-port fixes from eosio/eos release/2.2.x and removed use o…
66bdc38 
…f mutex for protecting session_set.
@heifner heifner mentioned this issue Jan 7, 2023
Merged
heifner added a commit that referenced this issue Jan 7, 2023
@heifner
GH-592 Use named_thread_pool instead of explicit thread, io_context, …
b2baf8d 
…etc. Old versions of boost do not have an acceptor that takes a strand in constructor.
@heifner
Copy link
Member Author

heifner commented Jan 9, 2023

Reporter indicated #624 fixes reported problem.

heifner added a commit that referenced this issue Jan 9, 2023
@heifner
GH-592 Merge GH-624 changes into split-ship-log branch.
e75b2dd
heifner added a commit that referenced this issue Jan 9, 2023
@heifner
GH-592 Debug log for request messages.
2970357
@heifner heifner mentioned this issue Jan 9, 2023
Merged
heifner added a commit that referenced this issue Jan 9, 2023
@heifner
GH-592 Use to_send_block_num
6df9b2b
heifner added a commit that referenced this issue Jan 9, 2023
@heifner
GH-592 Make sure ws::stream is destroyed on ship thread
9185b4f
heifner added a commit that referenced this issue Jan 11, 2023
@heifner
GH-592 Add ability to delay start of thread pool
25a7621
heifner added a commit that referenced this issue Jan 11, 2023
@heifner
GH-592 Delay start of thread pool until after listen
f0fad92
heifner added a commit that referenced this issue Jan 11, 2023
@heifner
GH-592 Add restart() needed to actually be able to restart io_context
5f29e56
@heifner heifner mentioned this issue Jan 11, 2023
Merged
heifner added a commit that referenced this issue Jan 18, 2023
@heifner
GH-592 Remove unused methods
1381a0f
heifner added a commit that referenced this issue Jan 18, 2023
@heifner
GH-592 WIP Fixing threading issues with state_history_log and sessions
b69a03f
heifner added a commit that referenced this issue Jan 18, 2023
@heifner
GH-592 Fix bad merge issue which was causing core dump
9aa3ffc
heifner added a commit that referenced this issue Jan 18, 2023
@heifner
GH-592 truncate/prune only called from other public methods that alre…
21bdaed 
…ady have a lock
heifner added a commit that referenced this issue Jan 19, 2023
@heifner
Merge branch 'main' into GH-592-ship-crash-split-ship-log
fe5e1f7
heifner added a commit that referenced this issue Jan 20, 2023
@heifner
Merge branch 'main' into GH-592-ship-crash-split-ship-log
3ed8ac7
heifner added a commit that referenced this issue Jan 25, 2023
@heifner
Merge remote-tracking branch 'origin/main' into GH-592-ship-crash-spl…
93173ce 
…it-ship-log
heifner added a commit that referenced this issue Jan 26, 2023
@heifner
GH-592 Fix spelling
6fda644
heifner added a commit that referenced this issue Jan 26, 2023
@heifner
Merge remote-tracking branch 'origin/main' into GH-592-ship-crash-spl…
0271a56 
…it-ship-log
heifner added a commit that referenced this issue Jan 26, 2023
@heifner
GH-592 Move log_* files only used by state_history into state_history…
23089d6 
… library.
heifner added a commit that referenced this issue Jan 27, 2023
@heifner
GH-592 Fix shutdown issue with session_test
34b8e11
heifner added a commit that referenced this issue Jan 27, 2023
@heifner
GH-592 Remove explicit close and just let websocket destructor take c…
f149ce6 
…are of it
heifner added a commit that referenced this issue Jan 30, 2023
@heifner
GH-592 Remove unneeded use of this->
33656e3
heifner added a commit that referenced this issue Jan 30, 2023
@heifner
GH-592 Increase to have more than one connection to ship
dc542bd
heifner added a commit that referenced this issue Jan 30, 2023
@heifner
GH-592 Add missing return
337b811
heifner added a commit that referenced this issue Jan 30, 2023
@heifner
Merge branch 'main' into GH-592-ship-crash-split-ship-log
0e5cd16
heifner added a commit that referenced this issue Jan 30, 2023
@heifner
GH-592 Minor cleanup from peer review comments
ed420c2
heifner added a commit that referenced this issue Jan 30, 2023
@heifner
Merge branch 'GH-592-ship-crash-split-ship-log' of https://github.com…
c137951 
…/AntelopeIO/leap into GH-592-ship-crash-split-ship-log
heifner added a commit that referenced this issue Jan 31, 2023
@heifner
GH-592 Move files back as they are used by block log splitting as well
24b5333
heifner added a commit that referenced this issue Feb 3, 2023
@heifner
Merge remote-tracking branch 'origin/main' into GH-592-ship-crash-spl…
7e1ef84 
…it-ship-log
huangminghuang added a commit that referenced this issue Feb 7, 2023
@huangminghuang
Merge branch 'main' into GH-592-ship-crash-split-ship-log
54ff3fe
huangminghuang added a commit that referenced this issue Feb 10, 2023
@huangminghuang
Merge branch 'huangminghuang/block_log' into GH-592-ship-crash-split-…
273a4a2 
…ship-log

# Conflicts:
#	libraries/chain/include/eosio/chain/log_catalog.hpp
#	libraries/chain/include/eosio/chain/log_data_base.hpp
#	libraries/chain/include/eosio/chain/log_index.hpp
heifner added a commit that referenced this issue Feb 20, 2023
@heifner
Merge branch 'huangminghuang/block_log' into GH-592-ship-crash-split-…
bd66489 
…ship-log
heifner added a commit that referenced this issue Feb 20, 2023
@heifner
Merge branch 'GH-592-ship-crash-split-ship-log' into GH-627-ship
16c2a92
heifner added a commit that referenced this issue Feb 21, 2023
@heifner
Merge branch 'huangminghuang/block_log' into GH-592-ship-crash-split-…
d12726c 
…ship-log
heifner added a commit that referenced this issue Feb 21, 2023
@heifner
Merge branch 'GH-592-ship-crash-split-ship-log' into GH-627-ship
5a94445
@heifner heifner mentioned this issue Feb 21, 2023
Merged
heifner added a commit that referenced this issue Mar 1, 2023
@heifner
Merge branch 'main' into GH-592-ship-crash-split-ship-log
248ed21
heifner added a commit that referenced this issue Mar 2, 2023
@heifner
GH-592 Add comment
6080511
huangminghuang added a commit that referenced this issue Mar 2, 2023
@huangminghuang
Merge pull request #628 from AntelopeIO/GH-592-ship-crash-split-ship-log
44f8c70 
Merge SHiP stability fixes & log splitting & memory usage improvements
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@heifner heifner

Labels
actionable bug Something isn't working 👍 lgtm OCI Work exclusive to OCI team
Projects
Team Backlog
Archived in project
Milestone
No milestone
3 participants
@heifner @stephenpdeos @enf-ci-bot