test: split simple into io/simple, run in parallel

[indutny]

Before:

[02:04|% 100|+ 786|-   0]: Done

After:

[01:44|% 100|+ 786|-   0]: Done

Fix #139

[indutny]

cc @bnoordhuis @piscisaureus

I'm still +1 in a favor of calling it parallel and sequential and moving more tests to parallel.

[indutny]

Here is a test split at the moment:

~/Code/indutny/node $ ls test/io/ | wc -l
     541
~/Code/indutny/node $ ls test/simple/ | wc -l
     232

I think it could be closer to 50%

[chrisdickinson]

In general, LGTM. A few thoughts:

1. I'd suggest putting README.md files in the two test directories ({simple,io}/) that state clearly what to put in each test dir, and why (if for no other reason than to have a place to point to when reviewing PRs!)
2. It would be interesting to see if we could parallelize more of the I/O tests by (ab)using common.PORT. If it turns out to be worthwhile, it'd probably be best to use the parallel / sequential nomenclature for the directory names.
3. This is probably just my inner Python programmer talking, but I am immediately suspicious of Python's threading; specifically how much parallelism it's actually delivering here. It could just be superstition talking, though :)

[bnoordhuis]

Is -J a typo? It's -j <jobs>, isn't it?

[indutny]

New test running time will make you cry, guys:

[00:32|% 100|+ 786|-   0]: Done

[indutny]

@bnoordhuis: btw, -J is not a typo.

[piscisaureus]

Windows result:

running 'python tools/test.py --mode=release simple message'
[00:04|% 100|+  19|-   0]: Done

IOW, only the message tests are being run.

[piscisaureus]

With a trivial fix (updating https://github.com/iojs/io.js/blob/165b70f146e163b82a09bb869463708516c08cf6/vcbuild.bat#L166):

[01:26|% 100|+ 785|-   1]: Done

That's great! I know it may not look so great to y'all spoiled unix users but that's 4 times faster than I'm used to.

Two small issues turned up.

1: I got a lot of these:

os.unlink() [Error 32] The process cannot access the file because it is being us
ed by another process: 'c:\\users\\bertbe~1\\appdata\\local\\temp\\tmpnwfzkp'

That's the test runner failing to delete a temp file. It doesn't cause any issues but it'd be nice to find out why it happens and if we should silence the error.

2: test-tls-server-verify timed out

That test used to time out a lot but I hadn't seen any failures recently.

[bnoordhuis]

Nice, LGTM. Just curious, is there a reason you had to change tests that bind to a random port?

[chrisdickinson]

LGTM. 👏

[bnoordhuis]

However... I do see a number of failures on the CI.

[indutny]

@bnoordhuis: Gosh, this is what I was fighting all day, and basically the reason why I was changing tests from .bind(0) to normal port. I wonder if I have missed some tests.

[indutny]

@bnoordhuis seems to be fixed by latest commit: https://jenkins-node-forward.nodesource.com/job/iojs+any-pr+multi/

[bnoordhuis]

LGTM, although I don't understand why changing from a random port to common.PORT would fix tests.

[indutny]

@bnoordhuis it looks like we was running the CI job in a wrong way. Also, the random port thing may conflict with common.PORT, as it gets a bit higher on higher id threads.

[indutny]

Landed in 0e19476 and b7d2478, thank you everyone!