two_node_network test case fails after transaction broadcast

[abitmore]

Bug Description

The two_node_network test case in app_test sometimes fails after message "Broadcasting tx", which indicates that the other node did not receive the transaction. Got some logs in #2374 (comment), quoted below.

No propagation:

2021-03-09T17:29:48.0288452Z Creating transfer tx
2021-03-09T17:29:48.0337249Z Pushing tx locally on db1
2021-03-09T17:29:48.0339773Z Broadcasting tx
2021-03-09T17:29:48.0356979Z 1788034ms p2p node.cpp:4682 broadcast ] broadcasting trx: {"trx":{"ref_block_num":0,"ref_block_prefix":0,"expiration":"2021-03-09T17:30:35","operations":[[37,{"fee":{"amount":0,"asset_id":"1.3.0"},"deposit_to_account":"1.2.17","balance_to_claim":"1.15.0","balance_owner_key":"BTS6MRyAjQq8ud7hVNYcfnVPJqcVpscN5So8BhtHuGYqET5GDW5CV","total_claimed":{"amount":"1000000000000000","asset_id":"1.3.0"}}],[0,{"fee":{"amount":2000000,"asset_id":"1.3.0"},"from":"1.2.17","to":"1.2.3","amount":{"amount":1000000,"asset_id":"1.3.0"},"extensions":[]}]],"extensions":[],"signatures":["205d73bd5e245ccf4ce3ffadb164fe77c322d9677a585835a85ac8193a38619e320e971e068e2475cd821d7934f8165505c86db194bcfc8b2c03f7abfdb1a785f8"]}}
2021-03-09T17:29:48.0365201Z 1788035ms p2p node.cpp:780 advertise_inventory_ ] beginning an iteration of advertise inventory
2021-03-09T17:29:48.0366973Z 1788035ms p2p node.cpp:794 advertise_inventory_ ] peer->peer_needs_sync_items_from_us: true
2021-03-09T17:29:48.0368501Z 1788035ms p2p peer_connection.cpp:497 clear_old_inventory ] Expiring old inventory for peer 127.0.0.1:40475: removing 0 items advertised to peer (0 left), and 0 advertised to us (0 left)

Normal propagation:

2021-03-09T17:21:10.0929382Z Creating transfer tx
2021-03-09T17:21:10.0975194Z Pushing tx locally on db1
2021-03-09T17:21:10.0988200Z 1270097ms p2p node.cpp:4682 broadcast ] broadcasting trx: {"trx":{"ref_block_num":0,"ref_block_prefix":0,"expiration":"2021-03-09T17:21:55","operations":[[37,{"fee":{"amount":0,"asset_id":"1.3.0"},"deposit_to_account":"1.2.17","balance_to_claim":"1.15.0","balance_owner_key":"BTS6MRyAjQq8ud7hVNYcfnVPJqcVpscN5So8BhtHuGYqET5GDW5CV","total_claimed":{"amount":"1000000000000000","asset_id":"1.3.0"}}],[0,{"fee":{"amount":2000000,"asset_id":"1.3.0"},"from":"1.2.17","to":"1.2.3","amount":{"amount":1000000,"asset_id":"1.3.0"},"extensions":[]}]],"extensions":[],"signatures":["200408e85aabcd7a46808f4d7e35436bbb7fbc15403052cc92ccae374ff43ca53779c59ffac4fafa36232381a15552830fbdad2ecf61b433ace79243234eeff4bd"]}}
2021-03-09T17:21:10.0997024Z 1270097ms p2p node.cpp:780 advertise_inventory_ ] beginning an iteration of advertise inventory
2021-03-09T17:21:10.1005865Z 1270097ms p2p node.cpp:794 advertise_inventory_ ] peer->peer_needs_sync_items_from_us: false
2021-03-09T17:21:10.1009207Z 1270097ms p2p node.cpp:802 advertise_inventory_ ] inventory_to_advertise: [{"item_type":1000,"item_hash":"f790d80149664991aabe2eaf2a601c139e8026d7"}]
2021-03-09T17:21:10.1012511Z Broadcasting tx
2021-03-09T17:21:10.1013603Z 1270097ms p2p node.cpp:816 advertise_inventory_ ] advertising item f790d80149664991aabe2eaf2a601c139e8026d7 to peer 127.0.0.1:36159
2021-03-09T17:21:10.1018860Z 1270097ms p2p node.cpp:829 advertise_inventory_ ] advertising 1 new item(s) of 1 type(s) to peer 127.0.0.1:36159

Impacts
Describe which portion(s) of BitShares Core may be impacted by this bug. Please tick at least one box.

• API (the application programming interface)
• Build (the build process or something prior to compiled code)
• CLI (the command line wallet)
• Deployment (the deployment process after building such as Docker, Travis, etc.)
• DEX (the Decentralized EXchange, market engine, etc.)
• P2P (the peer-to-peer network for transaction/block propagation)
• Performance (system or user efficiency, etc.)
• Protocol (the blockchain logic, consensus, validation, etc.)
• Security (the security of system or user data, etc.)
• UX (the User Experience)
• Other: testing

CORE TEAM TASK LIST

• Evaluate / Prioritize Bug Report
• Refine User Stories / Requirements
• Define Test Cases
• Design / Develop Solution
• Perform QA/Testing
• Update Documentation

[abitmore]

Fixed by #2377.