protractor-flake does not retry failed specs if the expectation is in a promise

[Klaas-Jan]

It seems that when you place an expectation in a promise in your test, the failed spec is not registered by protractor-flake as a spec it should retry.

Example:

it('should eventually do something', function (done) {
        var aPromise = somePage.somePromise();
        aPromise.then(function (value) {
           expect(value).toBe('something');
           done();
        });
   });

When value is not 'something', the test fails and counts as a failed test in the amount of failed tests at the end of the test run. But it is not being retried.

Example of a console log of such a situation (3 tests have expectations in promises):

...
11:07:56 [11:07:56] I/launcher - chrome #01-71 passed
11:07:56 [11:07:56] I/launcher - chrome #01-73 passed
11:07:56 [11:07:56] I/launcher - chrome #01-48 failed 3 test(s)
11:07:56 [11:07:56] I/launcher - chrome #01-68 passed
11:07:56 [11:07:56] I/launcher - chrome #01-77 passed
11:07:56 [11:07:56] I/launcher - chrome #01-79 passed
11:07:56 [11:07:56] I/launcher - chrome #01-81 failed 1 test(s)
11:07:56 [11:07:56] I/launcher - chrome #01-78 passed
11:07:56 [11:07:56] I/launcher - overall: 4 failed spec(s)
11:07:56 Closing report
11:07:56 [11:07:56] E/launcher - Process exited with error code 1
11:07:56 
11:07:56 Using standard to parse output
11:07:56 Re-running tests: test attempt 2
11:07:56 Re-running the following test files:
11:07:56 /home/jenkins/workspace/TA-3-test-chrome/viewtest/tests/referral/supplyPageSpec.js

It seems that when it is the only test that fails, all files are being retried, because it somehow notices there was a failure but does not have a list of failed tests. When another test fails (like the 'supplyPageSpec' above), only the supplyPageSpec is being retried and it 'forgets' about the other 3.

I solved this by moving the promises to a page object so I didn't have to place an expectation in a promise, but it seems to be a bug that the test is not being retried, am I right?

[NickTomlin]

This is because the stack trace for exceptions thrown in webdriver promises does not include the line of the file. Protractor flake takes a very simple approach to scrapping through the stacktraces and if this information isn't found it treats it as a re-run.

There isn't a great solve for this at the moment; I'd more than welcome a PR to make this better.

Two questions

• Can you share the output of the exception without using protractor flake?
• Can you share the options you are passing to protractor-flake?

[Klaas-Jan]

Output:

   The Management Basic Onboarding should
     ✓ start an onboarding tour
 .    ✓ continue the onboarding process on product page
 F    ✗ should move to the next step when navigating between pages
       - Expected 'basicManagementTour:2:0' to be 'basicManagementTour:1'.
           at /home/jenkins/workspace/TA-3-test-chrome/viewtest/tests/onboarding/ManagementBasicOnboardingSpec.js:37:31
           at ManagedPromise.invokeCallback_ (/usr/local/lib/node_modules/protractor/node_modules/selenium-webdriver/lib/promise.js:1379:14)
           at TaskQueue.execute_ (/usr/local/lib/node_modules/protractor/node_modules/selenium-webdriver/lib/promise.js:2913:14)
           at TaskQueue.executeNext_ (/usr/local/lib/node_modules/protractor/node_modules/selenium-webdriver/lib/promise.js:2896:21)
           at asyncRun (/usr/local/lib/node_modules/protractor/node_modules/selenium-webdriver/lib/promise.js:2775:27)
           at /usr/local/lib/node_modules/protractor/node_modules/selenium-webdriver/lib/promise.js:639:7
           at process._tickCallback (internal/process/next_tick.js:103:7)

Strange thing is ... I do see a filename to parse here.

Command line options:
protractor-flake --max-attempts=3 -- protractor.conf.js

This is because the stack trace for exceptions thrown in webdriver promises does not include the > line of the file. Protractor flake takes a very simple approach to scrapping through the
stacktraces and if this information isn't found it treats it as a re-run.

Maybe a quick fix would be that all the tests are being retried when the number of tests to retry don't match with the number of failed tests. I don't care if the tests have to take a while because they are all being retried, but failed tests not being retried while in the end the entire test suite is seen as 'green' is a bit dangerous ...

[NickTomlin]

Ah interesting; this output is from a different reporter than I am used to; unfortunately it still doesn't pass the criteria that the standard parser looks for.

Looking for passed files would be great but I don't know of a generic way to get which files passed and which did not (protractor doesn't provide this as far as I know). A custom reporter might solve this. Again, would love any ideas or a PR here.

I am going to work on writing the ability to make custom parsers for cases like this (since it seems like the spec file should always be the 1st at /stacktrace/path in your case. That way you can tailor it to your output and (hopefully) get the desired result. Does that sound acceptable?

[Klaas-Jan]

Sounds like a good idea.

[NickTomlin]

I have a pull request for custom parsers #45; I'd appreciate it if you'd take a look!

[NickTomlin]

Custom parsers have been released 9791183. I'm going to go ahead and close this for now. Feel free to reopen another issue if there is still a problem.