Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

After Launched and successfully upgraded BOS chain with 5 BP, LIB don't advance and stop at 900. #134

Closed
zhopen opened this issue Nov 3, 2019 · 8 comments
Closed

After Launched and successfully upgraded BOS chain with 5 BP, LIB don't advance and stop at 900. #134

zhopen opened this issue Nov 3, 2019 · 8 comments

Comments

@zhopen
Copy link

zhopen commented Nov 3, 2019

Docker image: boscore/bos:v3.0.2
Contracts: v3.0.2
BP: 5

After Launched and successfully upgraded BOS chain with 5 BP, LIB don't advance and stop at 900 foreever.

How do I fix it .

[root@192 bos.contracts]# cl get info
{
  "server_version": "b24a643a",
  "chain_id": "cf057bbfb72640471fd910bcb67639c22df9f92470936cddc1ade0e2f2e7dc4f",
  "head_block_num": 12005,
  "last_irreversible_block_num": 900,
  "last_irreversible_block_id": "00000384e8fe4e79400bb29e3736d7b14464c553d835e31ae7123d1dfd1fb9e1",
  "head_block_id": "00002ee5ea82b0f2923d44f3b7a2ddf638b13eab4d1f22aa4afd63f28271106c",
  "head_block_time": "2019-11-03T03:59:34.500",
  "head_block_producer": "prod1",
  "current_view": 0,
  "target_view": 8,
  "last_stable_checkpoint_block_num": 0,
  "virtual_block_cpu_limit": 200000000,
  "virtual_block_net_limit": 1048576000,
  "block_cpu_limit": 199900,
  "block_net_limit": 1048576,
  "server_version_string": "v3.0.2"
}

info  2019-11-03T03:59:15.987 thread-0  chain_plugin.cpp:389          plugin_initialize    ] initializing chain plugin
info  2019-11-03T03:59:15.995 thread-0  block_log.cpp:125             open                 ] Log is nonempty
info  2019-11-03T03:59:15.996 thread-0  block_log.cpp:152             open                 ] Index is nonempty
info  2019-11-03T03:59:16.083 thread-0  chain_plugin.cpp:740          plugin_initialize    ] include pbft controller...
info  2019-11-03T03:59:16.090 thread-0  pbft_database.cpp:56          pbft_database        ] checkpoint index size: 0
info  2019-11-03T03:59:16.091 thread-0  pbft.cpp:14                   pbft_controller      ] current view: 0
info  2019-11-03T03:59:16.093 thread-0  http_plugin.cpp:422           plugin_initialize    ] configured http to listen on 0.0.0.0:10001
info  2019-11-03T03:59:16.093 thread-0  net_plugin.cpp:3537           plugin_initialize    ] Initialize net plugin
info  2019-11-03T03:59:16.093 thread-0  net_plugin.cpp:3564           plugin_initialize    ] host: 0.0.0.0 port: 20001 
info  2019-11-03T03:59:16.093 thread-0  net_plugin.cpp:3636           plugin_initialize    ] my node_id is c75d551539bfe3ce1c6415a14d1f46b5ded52aab9ded8033b364be26b98045f1
info  2019-11-03T03:59:16.093 thread-0  pbft_plugin.cpp:176           plugin_initialize    ] Initialize pbft plugin
info  2019-11-03T03:59:16.093 thread-0  http_plugin.cpp:383           operator()           ] configured http with Access-Control-Allow-Headers : *
info  2019-11-03T03:59:16.093 thread-0  http_plugin.cpp:377           operator()           ] configured http with Access-Control-Allow-Origin: *
info  2019-11-03T03:59:16.093 thread-0  main.cpp:108                  main                 ] nodeos version v3.0.2
info  2019-11-03T03:59:16.093 thread-0  main.cpp:109                  main                 ] eosio root is /root/.local/share
info  2019-11-03T03:59:16.093 thread-0  main.cpp:110                  main                 ] nodeos using configuration file /eos/data-dir/config.ini
info  2019-11-03T03:59:16.093 thread-0  main.cpp:111                  main                 ] nodeos data directory is /eos/data-dir
info  2019-11-03T03:59:16.093 thread-0  chain_plugin.cpp:900          plugin_startup       ] starting chain in read/write mode
info  2019-11-03T03:59:16.093 thread-0  chain_plugin.cpp:904          plugin_startup       ] Blockchain started; head block is #11995, genesis timestamp is 2018-06-01T12:00:00.000
info  2019-11-03T03:59:16.093 thread-0  producer_plugin.cpp:765       plugin_startup       ] producer plugin:  plugin_startup() begin
info  2019-11-03T03:59:16.093 thread-0  producer_plugin.cpp:787       plugin_startup       ] Launching block production for 1 producers at 2019-11-03T03:59:16.093.
info  2019-11-03T03:59:16.093 thread-0  producer_plugin.cpp:799       plugin_startup       ] producer plugin:  plugin_startup() end
info  2019-11-03T03:59:16.093 thread-0  http_plugin.cpp:486           plugin_startup       ] start listening for http requests
info  2019-11-03T03:59:16.093 thread-0  chain_api_plugin.cpp:77       plugin_startup       ] starting chain_api_plugin
info  2019-11-03T03:59:16.093 thread-0  http_plugin.cpp:556           add_handler          ] add api url: /v1/chain/abi_bin_to_json
info  2019-11-03T03:59:16.093 thread-0  http_plugin.cpp:556           add_handler          ] add api url: /v1/chain/abi_json_to_bin
info  2019-11-03T03:59:16.093 thread-0  http_plugin.cpp:556           add_handler          ] add api url: /v1/chain/get_abi
info  2019-11-03T03:59:16.093 thread-0  http_plugin.cpp:556           add_handler          ] add api url: /v1/chain/get_account
info  2019-11-03T03:59:16.093 thread-0  http_plugin.cpp:556           add_handler          ] add api url: /v1/chain/get_block
info  2019-11-03T03:59:16.093 thread-0  http_plugin.cpp:556           add_handler          ] add api url: /v1/chain/get_block_header_state
info  2019-11-03T03:59:16.093 thread-0  http_plugin.cpp:556           add_handler          ] add api url: /v1/chain/get_code
info  2019-11-03T03:59:16.093 thread-0  http_plugin.cpp:556           add_handler          ] add api url: /v1/chain/get_code_hash
info  2019-11-03T03:59:16.093 thread-0  http_plugin.cpp:556           add_handler          ] add api url: /v1/chain/get_currency_balance
info  2019-11-03T03:59:16.093 thread-0  http_plugin.cpp:556           add_handler          ] add api url: /v1/chain/get_currency_stats
info  2019-11-03T03:59:16.093 thread-0  http_plugin.cpp:556           add_handler          ] add api url: /v1/chain/get_info
info  2019-11-03T03:59:16.093 thread-0  http_plugin.cpp:556           add_handler          ] add api url: /v1/chain/get_producer_schedule
info  2019-11-03T03:59:16.094 thread-0  http_plugin.cpp:556           add_handler          ] add api url: /v1/chain/get_producers
info  2019-11-03T03:59:16.094 thread-0  http_plugin.cpp:556           add_handler          ] add api url: /v1/chain/get_raw_abi
info  2019-11-03T03:59:16.094 thread-0  http_plugin.cpp:556           add_handler          ] add api url: /v1/chain/get_raw_code_and_abi
info  2019-11-03T03:59:16.094 thread-0  http_plugin.cpp:556           add_handler          ] add api url: /v1/chain/get_required_keys
info  2019-11-03T03:59:16.094 thread-0  http_plugin.cpp:556           add_handler          ] add api url: /v1/chain/get_scheduled_transactions
info  2019-11-03T03:59:16.094 thread-0  http_plugin.cpp:556           add_handler          ] add api url: /v1/chain/get_table_by_scope
info  2019-11-03T03:59:16.094 thread-0  http_plugin.cpp:556           add_handler          ] add api url: /v1/chain/get_table_rows
info  2019-11-03T03:59:16.094 thread-0  http_plugin.cpp:556           add_handler          ] add api url: /v1/chain/get_transaction_id
info  2019-11-03T03:59:16.094 thread-0  http_plugin.cpp:556           add_handler          ] add api url: /v1/chain/push_block
info  2019-11-03T03:59:16.094 thread-0  http_plugin.cpp:556           add_handler          ] add api url: /v1/chain/push_transaction
info  2019-11-03T03:59:16.094 thread-0  http_plugin.cpp:556           add_handler          ] add api url: /v1/chain/push_transactions
info  2019-11-03T03:59:16.094 thread-0  net_plugin.cpp:3658           plugin_startup       ] starting listener, max clients is 100
warn  2019-11-03T03:59:16.099 thread-0  net_plugin.cpp:855            connection           ] created connection to 192.168.195.139:20000
info  2019-11-03T03:59:16.099 thread-0  net_plugin.cpp:2140           connect              ] host: 192.168.195.139 port: 20000 
warn  2019-11-03T03:59:16.099 thread-0  net_plugin.cpp:855            connection           ] created connection to 192.168.195.139:20001
info  2019-11-03T03:59:16.099 thread-0  net_plugin.cpp:2140           connect              ] host: 192.168.195.139 port: 20001 
warn  2019-11-03T03:59:16.099 thread-0  net_plugin.cpp:855            connection           ] created connection to 192.168.195.139:20002
info  2019-11-03T03:59:16.099 thread-0  net_plugin.cpp:2140           connect              ] host: 192.168.195.139 port: 20002 
warn  2019-11-03T03:59:16.099 thread-0  net_plugin.cpp:855            connection           ] created connection to 192.168.195.139:20003
info  2019-11-03T03:59:16.099 thread-0  net_plugin.cpp:2140           connect              ] host: 192.168.195.139 port: 20003 
warn  2019-11-03T03:59:16.099 thread-0  net_plugin.cpp:855            connection           ] created connection to 192.168.195.139:20004
info  2019-11-03T03:59:16.099 thread-0  net_plugin.cpp:2140           connect              ] host: 192.168.195.139 port: 20004 
warn  2019-11-03T03:59:16.099 thread-0  net_plugin.cpp:855            connection           ] created connection to 192.168.195.139:20005
info  2019-11-03T03:59:16.099 thread-0  net_plugin.cpp:2140           connect              ] host: 192.168.195.139 port: 20005 
warn  2019-11-03T03:59:16.100 thread-0  net_plugin.cpp:881            connection           ] accepted network connection
error 2019-11-03T03:59:16.100 thread-0  net_plugin.cpp:2185           operator()           ] connection failed to 192.168.195.139:20003: Connection refused
error 2019-11-03T03:59:16.100 thread-0  net_plugin.cpp:2185           operator()           ] connection failed to 192.168.195.139:20004: Connection refused
error 2019-11-03T03:59:16.100 thread-0  net_plugin.cpp:2576           handle_message       ] Self connection detected. Closing connection
error 2019-11-03T03:59:16.100 thread-0  net_plugin.cpp:1280           operator()           ] sent a go away message: self connect, closing connection to connecting client
info  2019-11-03T03:59:16.100 thread-0  net_plugin.cpp:2672           handle_message       ] received a go away message from 192.168.195.139:20001, reason = self connect
warn  2019-11-03T03:59:16.197 thread-0  net_plugin.cpp:881            connection           ] accepted network connection
warn  2019-11-03T03:59:16.336 thread-0  net_plugin.cpp:881            connection           ] accepted network connection
warn  2019-11-03T03:59:16.850 thread-0  pbft_plugin.cpp:161           pbft_ready           ] 
******** BATCH-PBFT ENABLED ********
*                                  *
* --       The blockchain       -- *
* -  has successfully switched   - *
* -     into the new version     - *
* -        Please enjoy a        - *
* -      better performance!     - *
*                                  *
************************************

warn  2019-11-03T03:59:29.501 thread-0  transaction_context.cp:108    deadline_timer       ] Using polled checktime; deadline timer too inaccurate: min:6us max:63808us mean:6037us stddev:9168us
info  2019-11-03T03:59:30.002 thread-0  producer_plugin.cpp:1609      produce_block        ] Produced block 00002edcdf13bc87... #11996 @ 2019-11-03T03:59:30.000 signed by prod1 [trxs: 0, lib: 900, lscb: 0]
info  2019-11-03T03:59:30.504 thread-0  producer_plugin.cpp:1609      produce_block        ] Produced block 00002eddae4fbd2f... #11997 @ 2019-11-03T03:59:30.500 signed by prod1 [trxs: 0, lib: 900, lscb: 0]
info  2019-11-03T03:59:31.001 thread-0  producer_plugin.cpp:1609      produce_block        ] Produced block 00002edee94e1a37... #11998 @ 2019-11-03T03:59:31.000 signed by prod1 [trxs: 0, lib: 900, lscb: 0]
info  2019-11-03T03:59:31.501 thread-0  producer_plugin.cpp:1609      produce_block        ] Produced block 00002edf3340d15b... #11999 @ 2019-11-03T03:59:31.500 signed by prod1 [trxs: 0, lib: 900, lscb: 0]
info  2019-11-03T03:59:32.002 thread-0  producer_plugin.cpp:1609      produce_block        ] Produced block 00002ee03b6d7603... #12000 @ 2019-11-03T03:59:32.000 signed by prod1 [trxs: 0, lib: 900, lscb: 0]
info  2019-11-03T03:59:32.501 thread-0  producer_plugin.cpp:1609      produce_block        ] Produced block 00002ee11ad94990... #12001 @ 2019-11-03T03:59:32.500 signed by prod1 [trxs: 0, lib: 900, lscb: 0]
info  2019-11-03T03:59:33.001 thread-0  producer_plugin.cpp:1609      produce_block        ] Produced block 00002ee29e278dc0... #12002 @ 2019-11-03T03:59:33.000 signed by prod1 [trxs: 0, lib: 900, lscb: 0]
info  2019-11-03T03:59:33.501 thread-0  producer_plugin.cpp:1609      produce_block        ] Produced block 00002ee3e401a110... #12003 @ 2019-11-03T03:59:33.500 signed by prod1 [trxs: 0, lib: 900, lscb: 0]
info  2019-11-03T03:59:34.001 thread-0  producer_plugin.cpp:1609      produce_block        ] Produced block 00002ee47df2551d... #12004 @ 2019-11-03T03:59:34.000 signed by prod1 [trxs: 0, lib: 900, lscb: 0]
info  2019-11-03T03:59:34.501 thread-0  producer_plugin.cpp:1609      produce_block        ] Produced block 00002ee5ea82b0f2... #12005 @ 2019-11-03T03:59:34.500 signed by prod1 [trxs: 0, lib: 900, lscb: 0]
info  2019-11-03T03:59:35.000 thread-0  producer_plugin.cpp:1609      produce_block        ] Produced block 00002ee6357051e4... #12006 @ 2019-11-03T03:59:35.000 signed by prod1 [trxs: 0, lib: 900, lscb: 0]
info  2019-11-03T03:59:35.501 thread-0  producer_plugin.cpp:1609      produce_block        ] Produced block 00002ee7d1a226fc... #12007 @ 2019-11-03T03:59:35.500 signed by prod1 [trxs: 0, lib: 900, lscb: 0]
info  2019-11-03T03:59:36.001 thread-0  producer_plugin.cpp:353       on_incoming_block    ] Received block 6e832d5d4b5db195... #12008 @ 2019-11-03T03:59:36.000 signed by prod2 [trxs: 0, lib: 900, lscb: 0, latency: 1 ms]
info  2019-11-03T03:59:36.504 thread-0  producer_plugin.cpp:353       on_incoming_block    ] Received block 105c09ddab25f74b... #12009 @ 2019-11-03T03:59:36.500 signed by prod2 [trxs: 0, lib: 900, lscb: 0, latency: 4 ms]
info  2019-11-03T03:59:37.002 thread-0  producer_plugin.cpp:353       on_incoming_block    ] Received block fb76cb0d776adfbc... #12010 @ 2019-11-03T03:59:37.000 signed by prod2 [trxs: 0, lib: 900, lscb: 0, latency: 2 ms]
info  2019-11-03T03:59:37.503 thread-0  producer_plugin.cpp:353       on_incoming_block    ] Received block f7fd19f554c190b9... #12011 @ 2019-11-03T03:59:37.500 signed by prod2 [trxs: 0, lib: 900, lscb: 0, latency: 3 ms]
info  2019-11-03T03:59:38.003 thread-0  producer_plugin.cpp:353       on_incoming_block    ] Received block 356ee8a1ac975a93... #12012 @ 2019-11-03T03:59:38.000 signed by prod2 [trxs: 0, lib: 900, lscb: 0, latency: 3 ms]
info  2019-11-03T03:59:38.504 thread-0  producer_plugin.cpp:353       on_incoming_block    ] Received block 9198da48db8cd16a... #12013 @ 2019-11-03T03:59:38.500 signed by prod2 [trxs: 0, lib: 900, lscb: 0, latency: 4 ms]
info  2019-11-03T03:59:39.003 thread-0  producer_plugin.cpp:353       on_incoming_block    ] Received block 958053b18b99781b... #12014 @ 2019-11-03T03:59:39.000 signed by prod2 [trxs: 0, lib: 900, lscb: 0, latency: 3 ms]
info  2019-11-03T03:59:39.504 thread-0  producer_plugin.cpp:353       on_incoming_block    ] Received block 66469009e00bffd8... #12015 @ 2019-11-03T03:59:39.500 signed by prod2 [trxs: 0, lib: 900, lscb: 0, latency: 4 ms]
info  2019-11-03T03:59:40.004 thread-0  producer_plugin.cpp:353       on_incoming_block    ] Received block baf0e85498f56a1b... #12016 @ 2019-11-03T03:59:40.000 signed by prod2 [trxs: 0, lib: 900, lscb: 0, latency: 4 ms]
@oldcold
Copy link

oldcold commented Nov 5, 2019

The log snippet indicates that this node has just been restarted, can you pls provide a log history with more info? From your chain info, seems like it is waiting for consensus.
Did you attempt to change the bp schedule during the upgrade process? Do all active bps stay alive after the upgrade?

@zhopen
Copy link
Author

zhopen commented Nov 6, 2019 •
edited
Loading

@oldold
all of 5 BP stay alive before/after upgrading.
below is the log with more info

[root@localhost node]#  cl push action eosio setupgrade '{"up":{"target_block_num":1750}}' -p eosio

After upgrading to BPFT 
[root@localhost node]# cl get info
{
  "server_version": "b24a643a",
  "chain_id": "cf057bbfb72640471fd910bcb67639c22df9f92470936cddc1ade0e2f2e7dc4f",
  "head_block_num": 3873,
  "last_irreversible_block_num": 1759,
  "last_irreversible_block_id": "000006df67a66f93827480d2335dc452d40214d1665b7ffd7449311dce5c8999",
  "head_block_id": "00000f21d23768d17c796d6dbc1abc963c55b277221b8442c6c54c8c38c9ae3a",
  "head_block_time": "2019-11-06T05:36:30.500",
  "head_block_producer": "prod1",
  "current_view": 0,
  "target_view": 5,
  "last_stable_checkpoint_block_num": 0,
  "virtual_block_cpu_limit": 9602623,
  "virtual_block_net_limit": 50446224,
  "block_cpu_limit": 199900,
  "block_net_limit": 1048576,
  "server_version_string": "v3.0.2"
}

info  2019-11-06T05:18:47.504 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block f109aa545840e806... #1747 @ 2019-11-06T05:18:47.500 signed by prod3 [trxs: 0, lib: 1675, conf: 0, latency: 4 ms]
info  2019-11-06T05:18:48.002 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block 42a5e2c55cfa98ee... #1748 @ 2019-11-06T05:18:48.000 signed by prod4 [trxs: 0, lib: 1687, conf: 48, latency: 2 ms]
info  2019-11-06T05:18:48.503 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block f8856c6774bc545b... #1749 @ 2019-11-06T05:18:48.500 signed by prod4 [trxs: 0, lib: 1687, conf: 0, latency: 3 ms]
info  2019-11-06T05:18:49.003 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block a8dfa7917ab0e0ea... #1750 @ 2019-11-06T05:18:49.000 signed by prod4 [trxs: 0, lib: 1687, conf: 0, latency: 3 ms]
warn  2019-11-06T05:18:49.003 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:18:49.503 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:18:49.504 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block d24b56cb6d67a992... #1751 @ 2019-11-06T05:18:49.500 signed by prod4 [trxs: 0, lib: 1687, conf: 0, latency: 4 ms]
warn  2019-11-06T05:18:49.504 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:18:50.004 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:18:50.005 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block 3134d433877afa0d... #1752 @ 2019-11-06T05:18:50.000 signed by prod4 [trxs: 0, lib: 1687, conf: 0, latency: 5 ms]
warn  2019-11-06T05:18:50.005 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:18:50.501 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:18:50.502 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block 3719ce197eb13bd0... #1753 @ 2019-11-06T05:18:50.500 signed by prod4 [trxs: 0, lib: 1687, conf: 0, latency: 2 ms]
warn  2019-11-06T05:18:50.502 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:18:51.005 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:18:51.006 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block 521e361726d5e7cf... #1754 @ 2019-11-06T05:18:51.000 signed by prod4 [trxs: 0, lib: 1687, conf: 0, latency: 6 ms]
warn  2019-11-06T05:18:51.006 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:18:51.503 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:18:51.504 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block 67411a919259fcd6... #1755 @ 2019-11-06T05:18:51.500 signed by prod4 [trxs: 0, lib: 1687, conf: 0, latency: 4 ms]
warn  2019-11-06T05:18:51.504 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:18:52.001 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:18:52.001 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block e85aed2646a97f54... #1756 @ 2019-11-06T05:18:52.000 signed by prod4 [trxs: 0, lib: 1687, conf: 0, latency: 1 ms]
warn  2019-11-06T05:18:52.001 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:18:52.501 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:18:52.502 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block 5e957a9041bd80dc... #1757 @ 2019-11-06T05:18:52.500 signed by prod4 [trxs: 0, lib: 1687, conf: 0, latency: 2 ms]
warn  2019-11-06T05:18:52.502 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:18:53.002 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:18:53.004 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block fc431a90bc08b66b... #1758 @ 2019-11-06T05:18:53.000 signed by prod4 [trxs: 0, lib: 1687, conf: 0, latency: 4 ms]
warn  2019-11-06T05:18:53.004 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:18:53.501 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:18:53.502 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block 67a66f93827480d2... #1759 @ 2019-11-06T05:18:53.500 signed by prod4 [trxs: 0, lib: 1687, conf: 0, latency: 2 ms]
warn  2019-11-06T05:18:53.502 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:18:54.001 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:18:54.002 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block 3169913e44b8ae6c... #1760 @ 2019-11-06T05:18:54.000 signed by prod5 [trxs: 0, lib: 1699, conf: 48, latency: 2 ms]
warn  2019-11-06T05:18:54.002 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:18:54.505 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:18:54.505 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block 88d4e07167df3aec... #1761 @ 2019-11-06T05:18:54.500 signed by prod5 [trxs: 0, lib: 1699, conf: 0, latency: 5 ms]
warn  2019-11-06T05:18:54.505 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:18:55.002 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:18:55.003 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block bbed93cb4c7c76d6... #1762 @ 2019-11-06T05:18:55.000 signed by prod5 [trxs: 0, lib: 1699, conf: 0, latency: 3 ms]
warn  2019-11-06T05:18:55.003 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:18:55.501 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:18:55.502 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block f166bbbdd38bb77f... #1763 @ 2019-11-06T05:18:55.500 signed by prod5 [trxs: 0, lib: 1699, conf: 0, latency: 2 ms]
warn  2019-11-06T05:18:55.503 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:18:56.001 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:18:56.002 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block 8a1437bda4508bce... #1764 @ 2019-11-06T05:18:56.000 signed by prod5 [trxs: 0, lib: 1699, conf: 0, latency: 2 ms]
warn  2019-11-06T05:18:56.002 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:18:56.502 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:18:56.503 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block 02e5bfaf10d5c574... #1765 @ 2019-11-06T05:18:56.500 signed by prod5 [trxs: 0, lib: 1699, conf: 0, latency: 3 ms]
warn  2019-11-06T05:18:56.503 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:18:57.005 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:18:57.006 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block 71e5e8a0535a8666... #1766 @ 2019-11-06T05:18:57.000 signed by prod5 [trxs: 0, lib: 1699, conf: 0, latency: 6 ms]
warn  2019-11-06T05:18:57.006 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:18:57.502 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:18:57.503 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block 53b50f48fabc89f7... #1767 @ 2019-11-06T05:18:57.500 signed by prod5 [trxs: 0, lib: 1699, conf: 0, latency: 3 ms]
warn  2019-11-06T05:18:57.503 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:18:58.002 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:18:58.003 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block 2fee770df36cfa9e... #1768 @ 2019-11-06T05:18:58.000 signed by prod5 [trxs: 0, lib: 1699, conf: 0, latency: 3 ms]
warn  2019-11-06T05:18:58.003 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:18:58.503 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:18:58.504 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block b14fd654c7ef96e3... #1769 @ 2019-11-06T05:18:58.500 signed by prod5 [trxs: 0, lib: 1699, conf: 0, latency: 4 ms]
warn  2019-11-06T05:18:58.504 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:18:59.004 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:18:59.005 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block 20f2f5b99a8b7a7c... #1770 @ 2019-11-06T05:18:59.000 signed by prod5 [trxs: 0, lib: 1699, conf: 0, latency: 5 ms]
warn  2019-11-06T05:18:59.005 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:18:59.500 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:18:59.501 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:18:59.502 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block e78c6ad84bd66aa1... #1771 @ 2019-11-06T05:18:59.500 signed by prod5 [trxs: 0, lib: 1699, conf: 0, latency: 2 ms]
warn  2019-11-06T05:18:59.502 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:00.003 thread-0  producer_plugin.cpp:1614      produce_block        ] Produced block 000006ecb7cc1b21... #1772 @ 2019-11-06T05:19:00.000 signed by prod1 [trxs: 0, lib: 1711, confirmed: 48]
warn  2019-11-06T05:19:00.003 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:00.502 thread-0  producer_plugin.cpp:1614      produce_block        ] Produced block 000006ed5d238741... #1773 @ 2019-11-06T05:19:00.500 signed by prod1 [trxs: 0, lib: 1711, confirmed: 0]
warn  2019-11-06T05:19:00.502 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:01.001 thread-0  producer_plugin.cpp:1614      produce_block        ] Produced block 000006ee7371bb8d... #1774 @ 2019-11-06T05:19:01.000 signed by prod1 [trxs: 0, lib: 1711, confirmed: 0]
warn  2019-11-06T05:19:01.001 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:01.509 thread-0  producer_plugin.cpp:1614      produce_block        ] Produced block 000006efb0b60c02... #1775 @ 2019-11-06T05:19:01.500 signed by prod1 [trxs: 0, lib: 1711, confirmed: 0]
warn  2019-11-06T05:19:01.509 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:02.012 thread-0  producer_plugin.cpp:1614      produce_block        ] Produced block 000006f03db49d1f... #1776 @ 2019-11-06T05:19:02.000 signed by prod1 [trxs: 0, lib: 1711, confirmed: 0]
warn  2019-11-06T05:19:02.013 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:02.512 thread-0  producer_plugin.cpp:1614      produce_block        ] Produced block 000006f1d7ee24c7... #1777 @ 2019-11-06T05:19:02.500 signed by prod1 [trxs: 0, lib: 1711, confirmed: 0]
warn  2019-11-06T05:19:02.512 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:03.013 thread-0  producer_plugin.cpp:1614      produce_block        ] Produced block 000006f2a67001ee... #1778 @ 2019-11-06T05:19:03.000 signed by prod1 [trxs: 0, lib: 1711, confirmed: 0]
warn  2019-11-06T05:19:03.013 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:03.513 thread-0  producer_plugin.cpp:1614      produce_block        ] Produced block 000006f3d22e1435... #1779 @ 2019-11-06T05:19:03.500 signed by prod1 [trxs: 0, lib: 1711, confirmed: 0]
warn  2019-11-06T05:19:03.513 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:04.014 thread-0  producer_plugin.cpp:1614      produce_block        ] Produced block 000006f4d0caf90a... #1780 @ 2019-11-06T05:19:04.000 signed by prod1 [trxs: 0, lib: 1711, confirmed: 0]
warn  2019-11-06T05:19:04.014 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:04.504 thread-0  producer_plugin.cpp:1614      produce_block        ] Produced block 000006f56e7d705b... #1781 @ 2019-11-06T05:19:04.500 signed by prod1 [trxs: 0, lib: 1711, confirmed: 0]
warn  2019-11-06T05:19:04.504 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:05.012 thread-0  producer_plugin.cpp:1614      produce_block        ] Produced block 000006f62c761697... #1782 @ 2019-11-06T05:19:05.000 signed by prod1 [trxs: 0, lib: 1711, confirmed: 0]
warn  2019-11-06T05:19:05.012 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:05.511 thread-0  producer_plugin.cpp:1614      produce_block        ] Produced block 000006f70b2c71ea... #1783 @ 2019-11-06T05:19:05.500 signed by prod1 [trxs: 0, lib: 1711, confirmed: 0]
warn  2019-11-06T05:19:05.512 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:19:06.012 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:06.013 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block c211c77416396dab... #1784 @ 2019-11-06T05:19:06.000 signed by prod2 [trxs: 0, lib: 1723, conf: 48, latency: 13 ms]
warn  2019-11-06T05:19:06.013 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:19:06.517 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:06.518 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block 7d84611f08fd258d... #1785 @ 2019-11-06T05:19:06.500 signed by prod2 [trxs: 0, lib: 1723, conf: 0, latency: 18 ms]
warn  2019-11-06T05:19:06.518 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:19:07.014 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:07.014 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block 1bd648a1ac7a5da4... #1786 @ 2019-11-06T05:19:07.000 signed by prod2 [trxs: 0, lib: 1723, conf: 0, latency: 14 ms]
warn  2019-11-06T05:19:07.014 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:19:07.514 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:07.515 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block bc7806c4a2283f63... #1787 @ 2019-11-06T05:19:07.500 signed by prod2 [trxs: 0, lib: 1723, conf: 0, latency: 15 ms]
warn  2019-11-06T05:19:07.515 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:19:08.013 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:08.014 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block d7c3647a71f477da... #1788 @ 2019-11-06T05:19:08.000 signed by prod2 [trxs: 0, lib: 1723, conf: 0, latency: 14 ms]
warn  2019-11-06T05:19:08.014 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:19:08.531 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:08.531 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block 5c16935a13f93813... #1789 @ 2019-11-06T05:19:08.500 signed by prod2 [trxs: 0, lib: 1723, conf: 0, latency: 31 ms]
warn  2019-11-06T05:19:08.531 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:19:09.012 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:09.013 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block 750c072faad51938... #1790 @ 2019-11-06T05:19:09.000 signed by prod2 [trxs: 0, lib: 1723, conf: 0, latency: 13 ms]
warn  2019-11-06T05:19:09.013 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:19:09.512 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:09.513 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block ef8eee76dc133683... #1791 @ 2019-11-06T05:19:09.500 signed by prod2 [trxs: 0, lib: 1723, conf: 0, latency: 13 ms]
warn  2019-11-06T05:19:09.513 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:19:10.014 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:10.015 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block 596bafedf209398b... #1792 @ 2019-11-06T05:19:10.000 signed by prod2 [trxs: 0, lib: 1723, conf: 0, latency: 15 ms]
warn  2019-11-06T05:19:10.015 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:19:10.503 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:10.504 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block 87b5025c5d3d25ff... #1793 @ 2019-11-06T05:19:10.500 signed by prod2 [trxs: 0, lib: 1723, conf: 0, latency: 4 ms]
warn  2019-11-06T05:19:10.504 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:19:11.013 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:11.014 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block c8df1e9b130fbc11... #1794 @ 2019-11-06T05:19:11.000 signed by prod2 [trxs: 0, lib: 1723, conf: 0, latency: 14 ms]
warn  2019-11-06T05:19:11.014 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:19:11.512 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:11.512 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block 5683e21b362ed5e7... #1795 @ 2019-11-06T05:19:11.500 signed by prod2 [trxs: 0, lib: 1723, conf: 0, latency: 12 ms]
warn  2019-11-06T05:19:11.513 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:19:12.013 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:12.014 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block 85a3f62d68aa9469... #1796 @ 2019-11-06T05:19:12.000 signed by prod3 [trxs: 0, lib: 1735, conf: 48, latency: 14 ms]
warn  2019-11-06T05:19:12.014 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:19:12.502 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:12.502 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block e6aaa879e23c2087... #1797 @ 2019-11-06T05:19:12.500 signed by prod3 [trxs: 0, lib: 1735, conf: 0, latency: 2 ms]
warn  2019-11-06T05:19:12.503 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:19:13.012 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:13.013 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block d739839cd23600ea... #1798 @ 2019-11-06T05:19:13.000 signed by prod3 [trxs: 0, lib: 1735, conf: 0, latency: 13 ms]
warn  2019-11-06T05:19:13.013 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:19:13.512 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:13.513 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block 01943959cd910c73... #1799 @ 2019-11-06T05:19:13.500 signed by prod3 [trxs: 0, lib: 1735, conf: 0, latency: 13 ms]
warn  2019-11-06T05:19:13.513 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:19:14.012 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:14.013 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block ead3313461c50758... #1800 @ 2019-11-06T05:19:14.000 signed by prod3 [trxs: 0, lib: 1735, conf: 0, latency: 13 ms]
warn  2019-11-06T05:19:14.013 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:19:14.513 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:14.514 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block c41effc448c4c04b... #1801 @ 2019-11-06T05:19:14.500 signed by prod3 [trxs: 0, lib: 1735, conf: 0, latency: 14 ms]
warn  2019-11-06T05:19:14.514 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:19:15.013 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:15.013 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block 73678cef32fead31... #1802 @ 2019-11-06T05:19:15.000 signed by prod3 [trxs: 0, lib: 1735, conf: 0, latency: 13 ms]
warn  2019-11-06T05:19:15.013 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:19:15.513 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:15.514 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block 63d5de94a1d1424f... #1803 @ 2019-11-06T05:19:15.500 signed by prod3 [trxs: 0, lib: 1735, conf: 0, latency: 14 ms]
warn  2019-11-06T05:19:15.514 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:19:16.013 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:16.013 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block 25729c567902abb0... #1804 @ 2019-11-06T05:19:16.000 signed by prod3 [trxs: 0, lib: 1735, conf: 0, latency: 13 ms]
warn  2019-11-06T05:19:16.014 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:19:16.502 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:16.503 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block d130e46a6840ea7c... #1805 @ 2019-11-06T05:19:16.500 signed by prod3 [trxs: 0, lib: 1735, conf: 0, latency: 3 ms]
warn  2019-11-06T05:19:16.503 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:19:17.012 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:17.013 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block c9c50e6285587c49... #1806 @ 2019-11-06T05:19:17.000 signed by prod3 [trxs: 0, lib: 1735, conf: 0, latency: 13 ms]
warn  2019-11-06T05:19:17.013 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:19:17.512 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:17.513 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block 41a37d30397cf7bd... #1807 @ 2019-11-06T05:19:17.500 signed by prod3 [trxs: 0, lib: 1735, conf: 0, latency: 13 ms]
warn  2019-11-06T05:19:17.513 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:19:18.017 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:18.018 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block afdd197f372d8cf1... #1808 @ 2019-11-06T05:19:18.000 signed by prod4 [trxs: 0, lib: 1747, conf: 48, latency: 18 ms]
warn  2019-11-06T05:19:18.018 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:19:18.516 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:18.517 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block 9786da7371ffc98e... #1809 @ 2019-11-06T05:19:18.500 signed by prod4 [trxs: 0, lib: 1747, conf: 0, latency: 17 ms]
warn  2019-11-06T05:19:18.517 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:19:19.012 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:19.013 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block 60c047c139cfa32d... #1810 @ 2019-11-06T05:19:19.000 signed by prod4 [trxs: 0, lib: 1747, conf: 0, latency: 13 ms]
warn  2019-11-06T05:19:19.014 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:19:19.512 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:19.513 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block ac09f84a5241e3b2... #1811 @ 2019-11-06T05:19:19.500 signed by prod4 [trxs: 0, lib: 1747, conf: 0, latency: 13 ms]
warn  2019-11-06T05:19:19.513 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:19:20.012 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:20.014 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block b9ec9b783694a727... #1812 @ 2019-11-06T05:19:20.000 signed by prod4 [trxs: 0, lib: 1747, conf: 0, latency: 14 ms]
warn  2019-11-06T05:19:20.014 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:19:20.512 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:20.513 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block 1239f70434b0dc75... #1813 @ 2019-11-06T05:19:20.500 signed by prod4 [trxs: 0, lib: 1747, conf: 0, latency: 13 ms]
warn  2019-11-06T05:19:20.513 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:19:21.006 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:21.007 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block ca0f1bbeff0dcd66... #1814 @ 2019-11-06T05:19:21.000 signed by prod4 [trxs: 0, lib: 1747, conf: 0, latency: 7 ms]
warn  2019-11-06T05:19:21.007 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:19:21.506 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:21.507 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block 6561bb608f3ede49... #1815 @ 2019-11-06T05:19:21.500 signed by prod4 [trxs: 0, lib: 1747, conf: 0, latency: 7 ms]
warn  2019-11-06T05:19:21.507 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:19:22.013 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:22.013 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block 8848bae905290f1a... #1816 @ 2019-11-06T05:19:22.000 signed by prod4 [trxs: 0, lib: 1747, conf: 0, latency: 13 ms]
warn  2019-11-06T05:19:22.013 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:19:22.502 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:22.503 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block abb59c75c65d4f12... #1817 @ 2019-11-06T05:19:22.500 signed by prod4 [trxs: 0, lib: 1747, conf: 0, latency: 3 ms]
warn  2019-11-06T05:19:22.503 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:19:23.013 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:23.014 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block f9acc826ce5ab888... #1818 @ 2019-11-06T05:19:23.000 signed by prod4 [trxs: 0, lib: 1747, conf: 0, latency: 14 ms]
warn  2019-11-06T05:19:23.014 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:19:23.513 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:23.514 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block 6c11b36c5dd1a7ff... #1819 @ 2019-11-06T05:19:23.500 signed by prod4 [trxs: 0, lib: 1747, conf: 0, latency: 14 ms]
warn  2019-11-06T05:19:23.514 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:19:24.013 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:24.014 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block 9c97c1a4b764b074... #1820 @ 2019-11-06T05:19:24.000 signed by prod5 [trxs: 0, lib: 1759, conf: 48, latency: 14 ms]
warn  2019-11-06T05:19:24.014 thread-0  controller.cpp:443            update_pbft_status   ] pbft will be working after the block 1820
warn  2019-11-06T05:19:24.014 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
warn  2019-11-06T05:19:24.504 thread-0  controller.cpp:443            update_pbft_status   ] pbft will be working after the block 1820
warn  2019-11-06T05:19:24.504 thread-0  controller.cpp:1413           start_block          ] system is upgrading, no producer schedule promotion will happen until fully upgraded.
info  2019-11-06T05:19:24.505 thread-0  producer_plugin.cpp:359       on_incoming_block    ] Received block d1cbb214e34feffc... #1821 @ 2019-11-06T05:19:24.500 signed by prod5 [trxs: 0, lib: 1759, conf: 0, latency: 5 ms]
warn  2019-11-06T05:19:24.825 thread-0  pbft_plugin.cpp:161           pbft_ready           ] 
******** BATCH-PBFT ENABLED ********
*                                  *
* --       The blockchain       -- *
* -  has successfully switched   - *
* -     into the new version     - *
* -        Please enjoy a        - *
* -      better performance!     - *
*                                  *
************************************

info  2019-11-06T05:19:25.014 thread-0  producer_plugin.cpp:353       on_incoming_block    ] Received block 646d7efc83628823... #1822 @ 2019-11-06T05:19:25.000 signed by prod5 [trxs: 0, lib: 1759, lscb: 0, latency: 14 ms]
info  2019-11-06T05:19:25.514 thread-0  producer_plugin.cpp:353       on_incoming_block    ] Received block 602f4a4aef3b4d5c... #1823 @ 2019-11-06T05:19:25.500 signed by prod5 [trxs: 0, lib: 1759, lscb: 0, latency: 14 ms]
info  2019-11-06T05:19:26.014 thread-0  producer_plugin.cpp:353       on_incoming_block    ] Received block 19ea36dd8f3f5834... #1824 @ 2019-11-06T05:19:26.000 signed by prod5 [trxs: 0, lib: 1759, lscb: 0, latency: 14 ms]
info  2019-11-06T05:19:26.516 thread-0  producer_plugin.cpp:353       on_incoming_block    ] Received block 79a6ef35e2ef11e5... #1825 @ 2019-11-06T05:19:26.500 signed by prod5 [trxs: 0, lib: 1759, lscb: 0, latency: 16 ms]
info  2019-11-06T05:19:27.013 thread-0  producer_plugin.cpp:353       on_incoming_block    ] Received block 4064e6e4d2addbf6... #1826 @ 2019-11-06T05:19:27.000 signed by prod5 [trxs: 0, lib: 1759, lscb: 0, latency: 13 ms]
info  2019-11-06T05:19:27.513 thread-0  producer_plugin.cpp:353       on_incoming_block    ] Received block 8066c1ba87e296c7... #1827 @ 2019-11-06T05:19:27.500 signed by prod5 [trxs: 0, lib: 1759, lscb: 0, latency: 13 ms]
info  2019-11-06T05:19:28.012 thread-0  producer_plugin.cpp:353       on_incoming_block    ] Received block ace1caac24900f27... #1828 @ 2019-11-06T05:19:28.000 signed by prod5 [trxs: 0, lib: 1759, lscb: 0, latency: 12 ms]
info  2019-11-06T05:19:28.515 thread-0  producer_plugin.cpp:353       on_incoming_block    ] Received block 7d1a95884e6c5677... #1829 @ 2019-11-06T05:19:28.500 signed by prod5 [trxs: 0, lib: 1759, lscb: 0, latency: 15 ms]
info  2019-11-06T05:19:29.012 thread-0  producer_plugin.cpp:353       on_incoming_block    ] Received block 9cc2903784210496... #1830 @ 2019-11-06T05:19:29.000 signed by prod5 [trxs: 0, lib: 1759, lscb: 0, latency: 12 ms]
info  2019-11-06T05:19:29.513 thread-0  producer_plugin.cpp:353       on_incoming_block    ] Received block 0d2c827ce9955e91... #1831 @ 2019-11-06T05:19:29.500 signed by prod5 [trxs: 0, lib: 1759, lscb: 0, latency: 13 ms]
info  2019-11-06T05:19:30.012 thread-0  producer_plugin.cpp:1609      produce_block        ] Produced block 00000728e1ae386b... #1832 @ 2019-11-06T05:19:30.000 signed by prod1 [trxs: 0, lib: 1759, lscb: 0]
info  2019-11-06T05:19:30.500 thread-0  producer_plugin.cpp:1609      produce_block        ] Produced block 00000729de89f9b6... #1833 @ 2019-11-06T05:19:30.500 signed by prod1 [trxs: 0, lib: 1759, lscb: 0]
info  2019-11-06T05:19:31.012 thread-0  producer_plugin.cpp:1609      produce_block        ] Produced block 0000072a619c679b... #1834 @ 2019-11-06T05:19:31.000 signed by prod1 [trxs: 0, lib: 1759, lscb: 0]
info  2019-11-06T05:19:31.510 thread-0  producer_plugin.cpp:1609      produce_block        ] Produced block 0000072b28947c0d... #1835 @ 2019-11-06T05:19:31.500 signed by prod1 [trxs: 0, lib: 1759, lscb: 0]
info  2019-11-06T05:19:32.012 thread-0  producer_plugin.cpp:1609      produce_block        ] Produced block 0000072c40cb1056... #1836 @ 2019-11-06T05:19:32.000 signed by prod1 [trxs: 0, lib: 1759, lscb: 0]
info  2019-11-06T05:19:32.513 thread-0  producer_plugin.cpp:1609      produce_block        ] Produced block 0000072de3bbf463... #1837 @ 2019-11-06T05:19:32.500 signed by prod1 [trxs: 0, lib: 1759, lscb: 0]
info  2019-11-06T05:19:33.011 thread-0  producer_plugin.cpp:1609      produce_block        ] Produced block 0000072eadaed52f... #1838 @ 2019-11-06T05:19:33.000 signed by prod1 [trxs: 0, lib: 1759, lscb: 0]
info  2019-11-06T05:19:33.511 thread-0  producer_plugin.cpp:1609      produce_block        ] Produced block 0000072f3abab174... #1839 @ 2019-11-06T05:19:33.500 signed by prod1 [trxs: 0, lib: 1759, lscb: 0]
info  2019-11-06T05:19:34.012 thread-0  producer_plugin.cpp:1609      produce_block        ] Produced block 00000730c9897d18... #1840 @ 2019-11-06T05:19:34.000 signed by prod1 [trxs: 0, lib: 1759, lscb: 0]
info  2019-11-06T05:19:34.515 thread-0  producer_plugin.cpp:1609      produce_block        ] Produced block 0000073108874d90... #1841 @ 2019-11-06T05:19:34.500 signed by prod1 [trxs: 0, lib: 1759, lscb: 0]
info  2019-11-06T05:19:35.012 thread-0  producer_plugin.cpp:1609      produce_block        ] Produced block 000007324aa952fe... #1842 @ 2019-11-06T05:19:35.000 signed by prod1 [trxs: 0, lib: 1759, lscb: 0]
info  2019-11-06T05:19:35.504 thread-0  producer_plugin.cpp:1609      produce_block        ] Produced block 00000733ceb82bbe... #1843 @ 2019-11-06T05:19:35.500 signed by prod1 [trxs: 0, lib: 1759, lscb: 0]
info  2019-11-06T05:19:36.004 thread-0  producer_plugin.cpp:353       on_incoming_block    ] Received block 0b5676aca9a0ea05... #1844 @ 2019-11-06T05:19:36.000 signed by prod2 [trxs: 0, lib: 1759, lscb: 0, latency: 4 ms]
info  2019-11-06T05:19:36.503 thread-0  producer_plugin.cpp:353       on_incoming_block    ] Received block 80dc6bba13f3372d... #1845 @ 2019-11-06T05:19:36.500 signed by prod2 [trxs: 0, lib: 1759, lscb: 0, latency: 3 ms]

node1.log

@oldcold
Copy link

oldcold commented Nov 6, 2019 •
edited
Loading

That's weird..
Can you please do one thing: touching a file named logging.json to your config directory with the following content:
{ "includes": [], "appenders": [{ "name": "stderr", "type": "console", "args": { "stream": "std_error", "level_colors": [{ "level": "debug", "color": "green" },{ "level": "warn", "color": "brown" },{ "level": "error", "color": "red" } ] }, "enabled": true },{ "name": "stdout", "type": "console", "args": { "stream": "std_out", "level_colors": [{ "level": "debug", "color": "green" },{ "level": "warn", "color": "brown" },{ "level": "error", "color": "red" } ] }, "enabled": true } ], "loggers": [{ "name": "default", "level": "info", "enabled": true, "additivity": false, "appenders": [ "stderr" ] }, { "name":"net_plugin_impl", "level":"debug", "enabled":true, "additivity":false, "appenders": [ "stderr" ] } ] }
and then restart one of your bp nodes to see if you are able to see more logs.
By doing so, we can have more debug info which may help with this problem.

@zhopen
Copy link
Author

zhopen commented Nov 6, 2019

That's weird..
Can you please do one thing: touching a file named logging.json to your config directory with the following content:
{ "includes": [], "appenders": [{ "name": "stderr", "type": "console", "args": { "stream": "std_error", "level_colors": [{ "level": "debug", "color": "green" },{ "level": "warn", "color": "brown" },{ "level": "error", "color": "red" } ] }, "enabled": true },{ "name": "stdout", "type": "console", "args": { "stream": "std_out", "level_colors": [{ "level": "debug", "color": "green" },{ "level": "warn", "color": "brown" },{ "level": "error", "color": "red" } ] }, "enabled": true } ], "loggers": [{ "name": "default", "level": "info", "enabled": true, "additivity": false, "appenders": [ "stderr" ] }, { "name":"net_plugin_impl", "level":"debug", "enabled":true, "additivity":false, "appenders": [ "stderr" ] } ] }
and then restart one of your bp nodes to see if you are able to see more logs.
By doing so, we can have more debug info which may help with this problem.

@oldold
Get new log with logging.json:
node1-1.log

@oldcold
Copy link

oldcold commented Nov 7, 2019

@oldold
Get new log with logging.json:
node1-1.log

I didn't see any pbft messages transferring from the latter log, which system contract version did you use?

@zhopen
Copy link
Author

zhopen commented Nov 7, 2019 •
edited
Loading

@oldold
Get new log with logging.json:
node1-1.log

I didn't see any pbft messages transferring from the latter log, which system contract version did you use?

From hub.docker.com/boscore/bos
Docker image: boscore/bos:v3.0.2
System contract: https://github.com/boscore/bos.contract-prebuild tag:v3.0.2

@zhopen zhopen closed this as completed Nov 7, 2019
@zhopen
Copy link
Author

zhopen commented Nov 7, 2019 •
edited
Loading

can't use the same key pair for BPs.

@oldcold
Copy link

oldcold commented Nov 7, 2019 •
edited
Loading

can't use the same key pair for BPs.

In fact, we have done such tests several months ago, will reopen this issue and conduct some investigation here. Many thanks for your information.

@oldcold oldcold reopened this Nov 7, 2019
@oldcold oldcold added the bug Something isn't working label Nov 7, 2019
@Thaipanda Thaipanda removed the bug Something isn't working label Feb 12, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@zhopen @oldcold @Thaipanda