Add a nightly build to circleci which runs even the slow blockchain tests

[lithp]

Working off https://github.com/ethereum/py-evm/pull/1577/files#r240997519, it might be nice to run the SLOWEST_TESTS more often than never. I'm not sure if this works, I wasn't able to get it to run locally, circleci build --job py36-all-blockchain-tests kept failing with out of disk space errors.

Does anyone know of a better way to test it than merging and waiting for nightfall like this adorable little owl?

Cute Animal Picture

[carver]

Does anyone know of a better way to test it than merging and waiting for nightfall like this adorable little owl?

I feel about as knowledgeable about circle CI cron jobs as that owl looks.

Here is a potentially bad idea: what if we only ran the slowest tests in the nightly job? Even if we do that, I'm a bit worried that the tests will fail based on memory issues. I feel mildly confident that they will if we try to run everything plus the slow tests. Even if we only run slow tests, we might have to break them up further into smaller jobs to convince circle to run them.

[pipermerriam]

Anyone have experience setting up and running a Jenkins server? I think our best route to accomplish this would be having the devops team provisioning a more powerful machine that we can run these more intensive tests on (and maybe the benchmarks too since that machine is likely to have a more stable CPU profile.

[pipermerriam]

Might also be worth looking into whether https://github.com/features/actions can help us with this.

[veox]

Heads up!

PR #1579 pulls in tests that routinely consume 3 GiB of RAM when exec'd in a single process, or ~ 8 GiB at peak when allowed to run in parallel. CircleCI crashes for me when using Python 3.6.

These are now certainly listed in SLOWEST_TESTS. There are several (3?..), haven't pinpointed exactly which, but likely something from the top of these 5 pytest runs.

[veox]

(Unrelated to the above.)

Note that there are other tests that aren't run, which are not in SLOWEST_TESTS:

py-evm/tests/json-fixtures/test_blockchain.py

Lines 131 to 148 in 7e3b313

	def blockchain_fixture_mark_fn(fixture_path, fixture_name, fixture_fork):
	fixture_id = (fixture_path, fixture_name)
	if fixture_path.startswith('bcExploitTest'):
	return pytest.mark.skip("Exploit tests are slow")
	elif fixture_path == 'bcWalletTest/walletReorganizeOwners.json':
	return pytest.mark.skip("Wallet owner reorganization tests are slow")
	elif fixture_id in INCORRECT_UPSTREAM_TESTS:
	return pytest.mark.xfail(reason="Listed in INCORRECT_UPSTREAM_TESTS.")
	elif 'stTransactionTest/zeroSigTransa' in fixture_path:
	return pytest.mark.skip("EIP-86 not supported.")
	elif fixture_id in SLOWEST_TESTS:
	if should_run_slow_tests():
	return
	else:
	return pytest.mark.skip("Skipping slow test")
	elif 'stQuadraticComplexityTest' in fixture_path:
	return pytest.mark.skip("Skipping slow test")

Perhaps the function can be reworked to skip/xfail unsupported/incorrect, then check should_run_slow_tests(), then run if so (or not if not).

[lithp]

Opened #1606 to help reduce memory pressure. Hopefully it's enough to stop CircleCI from crashing!

[carver]

Let's see a run of this working (maybe by temporarily adding it to the main workflow) before we merge the change to the nightly.