R4R: Multi-seed parallel simulation

[cwgoes]

• Targeted PR against correct branch (see CONTRIBUTING.md)

• Linked to github-issue with discussion and accepted design OR link to spec that describes this work.

• Wrote tests

• Updated relevant documentation (docs/)

• Added entries in PENDING.md with issue #

• rereviewed Files changed in the github PR explorer

Ref #1924

---

For Admin Use:

• Added appropriate labels to PR (ex. wip, ready-for-review, docs)
• Reviewers Assigned
• Squashed all commits, uses message "Merge pull request #XYZ: [title]" (coding standards)

[ValarDragon]

While we're at it, can we also get make test_gaia_sim_fast to use different seeds each time as well? This would mean the randomized testing can be covering a wider surface. (Procedure on a bug find would be to test on develop as well)

[cwgoes]

While we're at it, can we also get make test_gaia_sim_fast to use different seeds each time as well? This would mean the randomized testing can be covering a wider surface. (Procedure on a bug find would be to test on develop as well)

Maybe. I like having deterministic failures, it makes CI easier to work with. Definitely in favor of running lots of random seeds before any release.

[cwgoes]

Seems some of the Amino serialization code is not goroutine-safe, or we're using singleton codes which aren't goroutine-safe. Presently running multiple GaiaApps in goroutines doesn't work (run make test_sim_gaia_full). I wonder if we want to fix this or not (can also run multi-seeds from a bash script or something).

[ValarDragon]

Not sure why we need this in go, running multiple in parallel within bash works fine on my machine.

[cwgoes]

Not sure why we need this in go, running multiple in parallel within bash works fine on my machine.

Immediately, probably not, but I could see it being advantageous in the future - running multiple instances of different Gaia apps during version upgrades, or maybe multiple "Apps" at once for a very fast IBC relay program.

[ValarDragon]

I think we can punt figuring out how to test multiple chains together safely to postlaunch. I think we should go with parallel calls within CI / bash for now.

Testing on my system as well, your analysis seems right, its probably a singleton map instance somewhere.

[codecov]

Codecov Report

Merging #2313 into develop will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           develop    #2313   +/-   ##
========================================
  Coverage    61.53%   61.53%           
========================================
  Files          122      122           
  Lines         7472     7472           
========================================
  Hits          4598     4598           
  Misses        2554     2554           
  Partials       320      320

[cwgoes]

I think we can punt figuring out how to test multiple chains together safely to postlaunch. I think we should go with parallel calls within CI / bash for now.

OK, I'm not entirely sure where the issue is so it might not be an easy fix - I still think we should find out though and make an informed decision.

Switched to a bash script for now.

[cwgoes]

This should be run as the "24-hour simulation" before cutting a release, possibly with more seeds / a higher number of blocks - and on a large multi-core machine.

[alexanderbez]

utACK -- just left a minor comment. Side note, are we not afraid we'll miss out on some potential bug discoveries by not using random seeds or will the fast sim use random seeds?

[alexanderbez]

can we break this command on multi-lines with \ to make it easier to read?

[cwgoes]

Yes, done.

[cwgoes]

Side note, are we not afraid we'll miss out on some potential bug discoveries by not using random seeds or will the fast sim use random seeds?

We should have a separate script which sequentially runs simulations with new random seeds and stops if it finds a failing seed - let's do that separately though - #2409.