Nodes need termination signal

[MaksymZavershynskyi]

Nodes need termination signal for testing.
See test_alphanet.rs, we cannot have more than one test per file because they consume CPU so much that if they run in parallel they timeout. We could introduce mutex to make sure we run only one at a time, however there is not way to stop a group of nodes.

[MaksymZavershynskyi]

Having this feature will also fix an issue with terminating our storage. Which causes errors.

Every now and then our tests complete successfully, but upon termination barf a SIGSEGV or SIGABRT. This behavior is similar to what Parity experiences in their storage code and might be an issue with rocksdb itself: openethereum/parity-ethereum#6213

From @bowenwang1996 's PR that he submitted (https://gitlab.com/nearprotocol/nearcore/-/jobs/177030804):

[2019-03-13T16:27:43Z INFO  coroutines::client_task] Producing block for account_id="bob.near", index 4
error: process didn't exit successfully: `/builds/nearprotocol/nearcore/target/debug/deps/alphanet-d7954231cbe86d8d` (signal: 11, SIGSEGV: invalid memory reference)

From my PR that I have not submitted yet (https://gitlab.com/nearprotocol/nearcore/-/jobs/177128839):

[2019-03-13T19:08:42Z INFO  client] Producing block index: 34, account_id="near.3", beacon hash = `3tJAhxHCYhKrmDX6ZLyvtMsXFvQ2SJZLtL18j3QEngFh`, shard hash = `9arbJJyKLopTbjCiQLQti1MYopxNBE78HNQBkNX6X2VZ`, #tx=0, #receipts=0
pure virtual method called
terminate called without an active exception
error: process didn't exit successfully: `/builds/nearprotocol/nearcore/target/debug/deps/test_alphanet-6ae30a2fdb0ed9aa` (signal: 6, SIGABRT: process abort signal)

P.S.
This is a second problem we have with using rocksdb. The previous one: facebook/rocksdb#3276

[mikhailOK]

#806

[MaksymZavershynskyi]

This issue is still happening. See: https://gitlab.com/nearprotocol/nearcore/-/jobs/197901603

test result: ok. 1 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out

     Running `/builds/nearprotocol/nearcore/target/debug/deps/test_alphanet_tps_regression-a5592fdfc61f596d`

running 1 test
test test_highload ... test test_highload has been running for over 60 seconds
Submitted transactions tps: [19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 188]Observed transactions tps: [15, 17, 18, 15, 25, 15, 20, 18, 18, 15, 22, 165]test test_highload ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

pure virtual method called
pure virtual method called
terminate called without an active exception
error: process didn't exit successfully: `/builds/nearprotocol/nearcore/target/debug/deps/test_alphanet_tps_regression-a5592fdfc61f596d` (signal: 6, SIGABRT: process abort signal)
Running after script...
$ find "${CACHE_ROOT}/target" -atime +2 -delete
ERROR: Job failed: exit code 1

[ilblackdragon]

With Actix we have a nice System::current().stop() that sends stop signal to all actors.