Fix TPS regression

[garious]

A performance regression slipped in since v0.5.0. It's most likely in the BankingStage, but we need to isolate it and see what can be done about it.

[sakridge]

@garious have you checked the multithreaded fix we put in for the balances table?

[garious]

@sakridge, no, which are you referring to?

[sakridge]

commit 1acd2aa
Author: Greg Fitzgerald garious@gmail.com
Date: Thu May 10 19:07:12 2018 -0600

Fix race condition in Accountant::apply_payment()

[garious]

When there's a preexisting to account, that patch is a no-op.

[garious]

@sakridge says there was no measurable gain from his key-sharding experiments, which aimed to minimize thread contention for the account memory.

[sakridge]

@garious @aeyakovenko
Tried to shard on bits in the public key for apply_debits -> apply_payment call flow, but I didn't manage to rework the register_entry_id() so it was still taking a write-lock. Also, apply_credits could be sharded.

Some results from #411 :

Macbook Pro 2016 15", i7-6920HQ (2.9ghz, 4c, 8t)

test	pre opt (ns)	with opt (ns)	speedup
bench_process_transaction	7,718,997	6,467,792	1.20x
bench_banking_stage_multi_accounts	611,069,540	68,183,547	9.0x
bench_banking_stage_single_from	237,442,138	58,952,941	4.0x

Intel i7-5930k Haswell-E (3.5ghz, 6c, 12t) 'newton'

test	pre opt (ns)	with opt (ns)	speedup
bench_process_transaction	6,707,948	3,093,161	2.2x
bench_banking_stage_multi_accounts	134,240,486	54,333,284	2.5x
bench_banking_stage_single_from	64,350,103	39,567,869	1.6x

AWS p2.8xlarge, Intel Xeon CPU E5-2686 v4 (2.2ghz, ?c, 32t)

test	pre opt (ns)	with opt (ns)	speedup
bench_process_transaction	9,070,171	4,074,513	2.2x
bench_banking_stage_multi_accounts	383,623,579	69,790,144	5.5x
bench_banking_stage_single_from	159,065,762	29,107,990	5.5x

Intel Xeon E5-2620 v4 (2.1ghz, 8c, 16t) 'sagan'

test	pre opt (ns)	with opt (ns)	speedup
bench_process_transaction	10,039,101	4,960,617	2x
bench_banking_stage_multi_accounts	202,239,418	88,446,154	2.3x
bench_banking_stage_single_from	95,234,199	49,350,139	1.9x

[garious]

@sakridge, can you post newton to sagan TPS numbers here so we can close this ticket? Just client-demo to leader. No validators.

[sakridge]

Highest TPS: 270133.73 sampling period 1s total transactions: 4947370 clients: 1

[sakridge]

@garious Are we good with that?

[garious]

Yes, perfect. Thanks so much for all this work!