Long-duration protocol exercise in simulation

[anorth]

Before production use, we should extensively exercise the protocol for long durations under different simulated chain and network conditions. This is an umbrella issue motivating a number of specific items to improve.

The simulator was mostly constructed to allow hand-crafting of specific situations to exercise protocol and coordination boundaries, mostly scoped to a single protocol instance. This is good, but limited to specific scenarios that we can think of. We should complement this with stochastic exercise over many instances under different macro conditions. The existing simulator can probably be used as a base, but the EC chain views and network need more complexity.

Some things to consider:

• More complexity in the EC chain, different view of head, different propagation of EC information to nodes, long-running forks etc (see Support divergent EC chain views in simulator #115)
• Network instability, periodic network halt, dropped messages
• Dynamic participation, some nodes refusing to participate
• Realistic network sizes of thousands of participants
• Highly uneven or more even power distribution

Some such exercises will be too slow to run in CI, but we should set up the capability to run multi-hour simulations on demand.

These exercises should be accompanied by some more metrics so we can measure things like the total network traffic, signature verification rate, count of instances that make progress vs basechain, etc (this might warrant a separate issue).

[BigLep]

@masih : I have added this to MX: Priority and sequencing TBD since I'm assuming this isn't needed anytime between now and shortly after f3 activation. Let me know if that is wrong.