24 failures when running all unit tests

[lkcampbell]

OS: Mac OSX Mavericks

Brackets Version: sprint 39 development build 0.39.0-0 (test 4d968b2).

Repro Steps:

1. Launch the Jasmine Spec Runner from Brackets
2. Select the All tab and click on All tests.

Expected Results:
All unit tests should pass.

Observed Results:
There are 24 total failures every time. Ten failures in the URL Code Hinting and fourteen failures in the JavaScript Code Hinting.

Other Notes:
If you run the URL Code Hinting and JavaScript Code Hinting suites individually, they all pass. If you select All unit tests from the Extensions tab, which also includes both these test suites, all tests pass.

[redmunds]

Lately, I have been seeing the same thing on Windows.

[JeffryBooher]

I get a ton of errors running integration tests all at once on Windows but I thought that was a known issue.

[njx]

What I generally see on Mac is that if I run "All", Brackets ends up using lots of memory and getting very slow, which I think starts to cause some tests to timeout. This is most likely because we have memory leaks when we open lots of separate windows in the Integration tests. So when I run unit tests, I generally run the Unit, Performance, and Extensions tabs, then quit and restart Brackets and run Integration, and that seems to usually work fine.

@lkcampbell does that sound like your symptoms?

In any case, we should spend some time to investigate what @redmunds and @JeffryBooher are seeing on Windows as well to see if that's the same problem, and figure out if there's something we can do about it, since it affects developer productivity.

Medium pri to @redmunds

[lkcampbell]

@njx, I have seen the slow performance problem you describe but I get this problem even after a fresh reboot. I can't see any delays but I still see the failures.

Most of the failures are Cannot get property 'foo' of null, or Cannot call method 'foo' of undefined.

[redmunds]

What I have noticed in the SpecRunner on Windows with Task Manager or Process Explorer, when I run a suite of tests (e.g. Unit, Integration) if I (1) "click All" versus (2) "clicking on each spec one at a time", the amount of memory used for (1) is roughly double the amount if memory used for (2).

@jasonsanjose Any ideas on why that happens?

Obviously, it would be nice to be able to click "All" and use half the memory. Also, none of the memory is released when you close the SpecRunner window -- you have to close Brackets.

I also notice that we're using Jasmine version 1, so we should probably upgrade to version 2. The APIs have changed (all waits*() functions have been removed for done() function), so it won't be an quick conversion.

[marcelgerber]

@njx I'm seeing the same issues @redmunds sees (we noticed that a few weeks ago and there's even another issue out there about the same problem).

[marcelgerber]

Ah, those are the more-or-less dupes I found: #7420, #4185

[lkcampbell]

@SAplayer, issue #7420 is not a duplicate for me because I can reproduce this issue but I can't reproduce #7420.

Issue #4185 may be relevant, however. If @redmunds is seeing more memory usage, it should be investigated, but, as I was saying to @njx above, I am not seeing sluggish behavior prior to this issue occurring. I have seen sluggish behavior after I run a few full unit test runs in a row, probably related to #7637 (comment). But if I reboot my machine and then immediately launch All Unit Tests, I still get this issue.

Maybe memory is the issue. Another thing to consider, maybe it is the unit tests occurring before these unit tests. If some unit tests aren't cleaning up after themselves, leaving behind functional side effects of some sort, it might affect these unit tests. Smaller subsets of unit test runs seem to avoid this issue. Maybe the problem unit tests aren't in the smaller subsets.

[lkcampbell]

Update on this issue.

As of Brackets sprint 40 development build 0.40.0-0 (master 4d9502f), the number of unit test failures is now up to 35. There are still 14 failures of JavaScript Code Hinting but now there are 21 failures with Url Code Hinting.

Note that these two sets of test suites are fired off very close to the end of the run after almost all of the other test suites have completed. Might still be a memory problem. Might be a functional side effect from another test suite that doesn't clean up after itself.

[redmunds]

The general feeling on the team is that we don't have enough unit tests and we need to add more, so until we find a major breakthrough in releasing memory after usage, I think this problem is going to continue to grow.

The workaround for now is to not run All/All tests, but to run All from each category (Unit, Integration, etc) at a time. But, even running a single category (most notably Integration) can hit the same problem, so I submitted pull request #7758 to split off the Live Development tests into a new category. Note that this PR also moves some "integration" type tests out of the Unit category to make those tests run faster.

[redmunds]

We keep adding unit tests, so it's impossible as far as I know to run All/All unit tests at this point. I created a wiki page with Tips and Tricks for running Brackets unit tests to help people work around this issue.

[lkcampbell]

I'm closing this issue as resolved by documentation per @redmunds comment above.