Add regression test harness upgrade proposal.

[jgiszczak]

No description provided.

[spoonincode]

This many not be portable. I don't think macOS routes all of 127.0/8 by default, for example. How difficult will it be to have a fallback for such a platform that still executes sequentially?

[jgiszczak]

I wasn't aware macOS is not RFC 3330 compliant. macOS is not a supported platform. If and when we support it, this: sudo ifconfig lo0 alias 127.0.0.2 up can be added to the test harness when it's running on macOS. The Python platform module can be used to identify macOS as needed.

[spoonincode]

So are you saying for macos you can use a strategy that maps all IP & ports to 127.0.0.2:x?

[jgiszczak]

Sorry, I wasn't clear. Variations of that sudo would be used for all IPs needed in the subnet. 127.0.0.x.

[spoonincode]

What will happen when a user runs ctest in this case?

[jgiszczak]

When a macOS user runs ctest they will get one password prompt on the command line the first time any test runs which requires the loopback addresses. All tests will check for their presence and silently use them if they're available, which will avoid additional password prompts that might otherwise arise because the sudo timeout has been reached before the entire suite of tests has completed.

[spoonincode]

How/where will the dashboard be created?

[jgiszczak]

Ted requested it. Whether or not its data source is a Github action and where it would be hosted is yet to be determined.

[spoonincode]

I do have one special request as part of the refactor: remove usage of curl in the python tests. There are a few usages of it currently, but it seems needless since python has built in HTTP client library

[stephenpdeos]

A few notes from yesterday's discussion

Clarifications:

• Most of this work is just a refactor of existing code written in python. The only true rewrite considered within this proposal is for eosio-launcher
• Containerization via docker to help test run in parallel is not within scope because of its limited use by our assumed developer base
• There are no major concern with how this scope of work intersects with @spoonincode 's ongoing efforts with build process, there will just need to be some degree of coordination and communication
• The existing dfuse related tests were not considered within scope of this proposal, but warrant further discussion

Decisions:

• Python is the correct language for this harness, namely because of its widespread use, flexibility, and that the codebase proposed to be refactored is already written in python
• There should be additional robustness around transaction lifecycle added to this proposal
• The supporting functions for setup and teardown of should be modularized out so it can be used across more use cases for various client test environments. This may not be required for an initial milestone for this initiative

[matthewdarwin]

Another point that was discussed is that the framework used to spin up a node(cluster) should be usable by downstream projects like eosjs.

[spoonincode]

Yes, good to capture that requirement. It seems like the python script/scripts/module ought to be included in either the leap or leap-dev package.

[stephenpdeos]

Lgtm