Protractor-flake running again the tests that have passed #54

rafaelcs · 2017-04-08T03:12:57Z

How can I configure to run only those tests that failed? Currentlyits running even the tests that passed the first time

wswebcreation · 2017-04-08T07:20:33Z

What do you exactly mean? Does it run alle the spec / step files again, including the spec / step-files that hold no failed tests?

protractor-flake can only run a spec / stepfile which holds at least 1 failed test. It can not specifically run the failed test.This due to the log parser that will check which test from which file failed, it will then collect all the files with failed specs and run them again, leaving out the spec / step-files that don't hold any failed tests.

rafaelcs · 2017-04-08T07:24:04Z

I mean that, for example, I have spec1.js and spec2.js...if only the spec1.js is failing, why spec2.js run again too? Its my problem Em 8 de abr de 2017 4:20 AM, "wswebcreation" <notifications@github.com> escreveu:

…

Hi @rafaelcs <https://github.com/rafaelcs> , What do you exactly mean? Does it run alle the spec / step files again, including the spec / step-files that hold no failed tests? protractor-flake can only run a spec / stepfile which holds at least 1 failed test. It can not specifically run the failed test.This due to the log parser that will check which test from which file failed, it will then collect all the files with failed specs and run them again, leaving out the spec / step-files that don't hold any failed tests. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#54 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHvNk7hgQHZ_DIiIHxtaWLKr4jAPdaeJks5rtzVBgaJpZM4M3ktI> .

wswebcreation · 2017-04-08T07:28:26Z

Can you provide us the following info:

configuration, how do you run protractor-flake
logging

Tnx in advance

rafaelcs · 2017-04-17T21:38:50Z

Sorry for delay @wswebcreation

My flake file:

#!/usr/bin/env node

var protractorFlake = require('protractor-flake');
// skip first two passed args (node and self)
var protractorArgs = process.argv.splice(2);

protractorFlake({
    protractorPath: 'node_modules/.bin/protractor',
    maxAttempts: 2,
    parser: 'standard',
    nodeBin: 'node',
    protractorArgs: protractorArgs
}, function(status, output) {
    process.exit(status);
});

My conf.js


exports.config = {

    directConnect: true,
    
    specs: ['tests/UI/specs/*.js'],
    seleniumAddress: 'http://localhost:4445/wd/hub',
    framework: 'jasmine2',

    onPrepare: function(){
        // set implicit wait times in ms...
        browser.manage().timeouts().implicitlyWait(5000);

        browser.ignoreSynchronization = true;
        
        // set browser size...
        browser.manage().window().maximize();

        // better jasmine 2 reports...
        let SpecReporter = require('jasmine-spec-reporter').SpecReporter;
        jasmine.getEnv().addReporter(new SpecReporter({displayStacktrace: 'spec'}));
    },

    capabilities: {
		browserName: 'chrome',
        restartBrowserBetweenTests: true
	},

    jasmineNodeOpts: {
        showColors: true,
        displayStacktrace: true,
        displaySpecDuration: true,
        // overrides jasmine's print method to report dot syntax for custom reports
        print: function () {},
        defaultTimeoutInterval: 999999
    }
};

On this screen, you can see that only two tests failed

But when the tests ran again, all the specs were executed instead of just running the specs that failed

I'm running the tests using ./flake conf.js

That's my problem so far

wswebcreation · 2017-04-18T04:50:38Z

Hi @rafaelcs,

Is this the only trace you have from your tests, because I think the specreporter is the problem here. What protractor-flake does is that it:

checks the exitcode after all tests are ran
reads the log and parses it.

When I look at you log there is no logging that matches this regex of the standard parser.

I think there are 3 options for you:

write a custom parser to use and parse your output
change the logging of the specreporter so it will output more logging that will match the parser
write a "hook" that will print the filename after a test has failed so it will be picked up by the standard / custom parser

Hope it helps

Grtz
Wim

surendraJ · 2017-06-20T02:57:52Z

Recently I am also getting this error.

98 specs, 1 failure
Finished in 446.454 seconds

[21:37:14] I/launcher - 0 instance(s) of WebDriver still running
[21:37:14] I/launcher - chrome #1 failed 1 test(s)
[21:37:14] I/launcher - overall: 1 failed spec(s)
Closing report
[21:37:14] E/launcher - Process exited with error code 1

Using standard to parse output
Re-running tests: test attempt 2

Tests failed but no specs were found. All specs will be run again.

This is very frustrating as my all specs are running again even there is only one spec failure. Here is my configuration....
===gulpfile.js ===

gulp.task('e2e', function(){
    return protractorFlake({
        protractorPath: 'node_modules/protractor/bin/protractor',
        maxAttempts: 3,
        parser: 'standard',
        // expects node to be in path
        // set this to wherever the node bin is located
        nodeBin: 'node',
        // set color to one of the colors available at 'chalk' - https://github.com/chalk/ansi-styles#colors
        color: 'magenta',
        protractorArgs: ["e2e/conf/protractor.conf.js"]
    }, function (status, output) {
        process.exit(status);
    });
});

==== package.json=====

"jasmine-core": "^2.6.0",
"protractor": "^5.1.1",
protractor-jasmine2-screenshot-reporter": "^0.3.5",    
"protractor-flake": "2.5.1"

===protractor.conf.js=====

var HtmlScreenshotReporter = require('protractor-jasmine2-screenshot-reporter');
var htmlReporter = new HtmlScreenshotReporter({
	 	dest: 'e2e/testreports', 
	    filename: 'e2eReport.html', 
	    reportOnlyFailedSpecs: false,
		captureOnlyFailedSpecs: true, 
		showSummary: true,
		showQuickLinks: true
});
exports.config = {
     .......
    specs: ['../spec/*.js'],
    framework:'jasmine2',
    multiCapabilities: [
        {browserName: 'chrome'}
    ],
    directConnect: true,

    onPrepare: function() {
       .......
        jasmine.getEnv().addReporter(htmlReporter);
        var jasmineReporters = require('jasmine-reporters');
        jasmine.getEnv().addReporter(new jasmineReporters.JUnitXmlReporter({
            consolidateAll: true,
            savePath: 'e2e/testreports',
            filePrefix: 'e2eReport'
        }));
    },
..........

And I will run gulp task to call protractor flake. Would you please check this one what is happening?

wswebcreation · 2017-06-20T03:56:47Z

Hi @surendraJ

Can you please provide your logging so we can see what protractor-flake should parse?

Tnx in advance

surendraJ · 2017-06-20T04:07:29Z

@wswebcreation How can I generate the log? Is there any configuration do I need to fix to see the log?

wswebcreation · 2017-06-20T04:17:40Z

@surendraJ It's you console log / output from your protractor tests

surendraJ · 2017-06-20T04:27:26Z

@wswebcreation here is my log I am not sure it would be helpful or not...

Stack:
    Error: Failed expectation
        at C:\Jenkins\workspace\mypro\e2e\spec\landing-page.spec.js:39:31
        at ManagedPromise.invokeCallback_ (C:\Jenkins\workspace\mypro\node_modules\selenium-webdriver\lib\promise.js:1366:14)
        at TaskQueue.execute_ (C:\Jenkins\workspace\mypro\node_modules\selenium-webdriver\lib\promise.js:2970:14)
        at TaskQueue.executeNext_ (C:\Jenkins\workspace\mypro\node_modules\selenium-webdriver\lib\promise.js:2953:27)
        at asyncRun (C:\Jenkins\workspace\mypro\node_modules\selenium-webdriver\lib\promise.js:2813:27)
        at C:\Jenkins\workspace\mypro\node_modules\selenium-webdriver\lib\promise.js:676:7
        at process._tickCallback (internal/process/next_tick.js:103:7)
		
98 specs, 1 failure
Finished in 446.454 seconds

[21:37:14] I/launcher - 0 instance(s) of WebDriver still running
[21:37:14] I/launcher - chrome #01 failed 1 test(s)
[21:37:14] I/launcher - overall: 1 failed spec(s)
Closing report
[21:37:14] E/launcher - Process exited with error code 1

Using standard to parse output
Re-running tests: test attempt 2

Tests failed but no specs were found. All specs will be run again.

wswebcreation · 2017-06-20T04:31:22Z

@surendraJ Is this the full log?

We need to the full log because, as also mentioned above, protractor-flake reads the output from protractor to determine which steps-files failed. Based on your logging I can check what goes wrong, so it would be nice if you can provide the complete output from your tests.

surendraJ · 2017-06-20T04:42:26Z

@wswebcreation It's very huge log file so all are the success log before this one and failure message and again start 2nd attempt...

Failures:
1) #Landing page - [landing-page.spec.js]  should show autocomplete with first entry of ZIP, 52 when entered Zip in location field
  Message:
[31m    Expected [ 'ZIP, 51', 'ZIP, 52', 'ZIP, 53', 'ZIP, 54', 'ZIP, 55', 'ZIP, 56' ] to contain 'ZIP, 61'.�[0m
  Stack:
    Error: Failed expectation
        at C:\Jenkins\workspace\mypro\e2e\spec\landing-page.spec.js:38:31
        at ManagedPromise.invokeCallback_ (C:\Jenkins\workspace\mypro\node_modules\selenium-webdriver\lib\promise.js:1366:14)
        at TaskQueue.execute_ (C:\Jenkins\workspace\mypro\node_modules\selenium-webdriver\lib\promise.js:2970:14)
        at TaskQueue.executeNext_ (C:\Jenkins\workspace\mypro\node_modules\selenium-webdriver\lib\promise.js:2953:27)
        at asyncRun (C:\Jenkins\workspace\mypro\node_modules\selenium-webdriver\lib\promise.js:2813:27)
        at C:\Jenkins\workspace\mypro\node_modules\selenium-webdriver\lib\promise.js:676:7
        at process._tickCallback (internal/process/next_tick.js:103:7)
  Message:
�[31m    Expected [ 'ZIP, 51', 'ZIP, 52', 'ZIP, 53', 'ZIP, 54', 'ZIP, 55', 'ZIP, 56' ] to contain 'ZIP, 61'.�[0m
  Stack:
    Error: Failed expectation
        at C:\Jenkins\workspace\mypro\e2e\spec\landing-page.spec.js:39:31
        at ManagedPromise.invokeCallback_ (C:\Jenkins\workspace\mypro\node_modules\selenium-webdriver\lib\promise.js:1366:14)
        at TaskQueue.execute_ (C:\Jenkins\workspace\mypro\node_modules\selenium-webdriver\lib\promise.js:2970:14)
        at TaskQueue.executeNext_ (C:\Jenkins\workspace\mypro\node_modules\selenium-webdriver\lib\promise.js:2953:27)
        at asyncRun (C:\Jenkins\workspace\mypro\node_modules\selenium-webdriver\lib\promise.js:2813:27)
        at C:\Jenkins\workspace\mypro\node_modules\selenium-webdriver\lib\promise.js:676:7
        at process._tickCallback (internal/process/next_tick.js:103:7)

98 specs, 1 failure
Finished in 446.454 seconds

[21:37:14] I/launcher - 0 instance(s) of WebDriver still running
[21:37:14] I/launcher - chrome #01 failed 1 test(s)
[21:37:14] I/launcher - overall: 1 failed spec(s)
Closing report
[21:37:14] E/launcher - Process exited with error code 1

Using standard to parse output
Re-running tests: test attempt 2

Tests failed but no specs were found. All specs will be run again.

Report destination:   e2e\testreports\e2eReport.html
[21:37:14] W/driverProviders - Using driver provider directConnect, but also found extra driver provider parameter(s): seleniumServerJar
[21:37:14] I/launcher - Running 1 instances of WebDriver
[21:37:14] I/direct - Using ChromeDriver directly...

wswebcreation · 2017-06-20T04:49:40Z

If this is the only logging for the failure then the standard / multi parser will not detect any failures in your logging. This means it will rerun all your specs again.

My assumption is that the jasmine-reporter is "cleaning" up the log the shown in the output. In my opinion this is the same problem @rafaelcs has and can only be fixed with a "customer" parser.

@NickTomlin , how do you think about this?

wpalkuri · 2017-08-01T09:29:22Z

Hi @wswebcreation -
@NickTomlin - fyi
Please see my total log as below. You should see out of 3 tests, 2 tests have passed and 1 test failed. Expecting protractor-flake to re run ONLY the failed test in 2nd attempt. However, it runs all 3 tests again. Can you please suggest a solution or let me know if i missed something.

Thanks in advance.

C:\Users\wilson.palkuri\Downloads\protractor-example-master\protractor-example-master>protractor-flake --maxAttempts=2 -- conf.js
[14:53:34] I/launcher - Running 1 instances of WebDriver
[14:53:45] I/testLogger -
------------------------------------

[14:53:45] I/testLogger - [chrome #01-0] PID: 11920
[chrome #01-0] Specs: C:\Users\wilson.palkuri\Downloads\protractor-example-master\protractor-example-master\specs\friendSpec.js
[chrome #01-0]
[chrome #01-0] [14:53:38] I/direct - Using ChromeDriver directly...
[chrome #01-0] [14:53:45] I/runnerCli - Cannot find module 'chance'

[14:53:45] I/testLogger -

[14:53:45] E/launcher - Runner process exited unexpectedly with error code: 1
[14:53:46] I/launcher - 1 instance(s) of WebDriver still running
.[14:53:56] I/testLogger -
------------------------------------

[14:53:56] I/testLogger - [chrome #01-1] PID: 10200
[chrome #01-1] Specs: C:\Users\wilson.palkuri\Downloads\protractor-example-master\protractor-example-master\specs\nonAngularLoginSpec.js
[chrome #01-1]
[chrome #01-1] [14:53:49] I/direct - Using ChromeDriver directly...
[chrome #01-1] Jasmine started
[chrome #01-1] No Angular PASS
[chrome #01-1]
[chrome #01-1]   non-angular login
[chrome #01-1]     √ should goto friend pages on successful login
[chrome #01-1]
[chrome #01-1] Executed 1 of 1 spec SUCCESS in 1 sec.

[14:53:56] I/testLogger -

[14:53:56] I/launcher - 1 instance(s) of WebDriver still running
..[14:54:22] I/testLogger -
------------------------------------

[14:54:22] I/testLogger - [chrome #01-2] PID: 152
[chrome #01-2] Specs: C:\Users\wilson.palkuri\Downloads\protractor-example-master\protractor-example-master\specs\qsSpec.js
[chrome #01-2]
[chrome #01-2] [14:54:00] I/direct - Using ChromeDriver directly...
[chrome #01-2] Jasmine started
[chrome #01-2]
[chrome #01-2]   Quality Shepherd blog
[chrome #01-2]     √ should display 5 posts per page
[chrome #01-2]     √ should return search results
[chrome #01-2]
[chrome #01-2] Executed 2 of 2 specs SUCCESS in 17 secs.

[14:54:22] I/testLogger -

[14:54:22] I/launcher - 0 instance(s) of WebDriver still running
[14:54:22] I/launcher - chrome #01-0 failed with exit code: 1
[14:54:22] I/launcher - chrome #01-1 passed
[14:54:22] I/launcher - chrome #01-2 passed
[14:54:22] I/launcher - overall: 1 process(es) failed to complete
[14:54:22] E/launcher - Process exited with error code 100

Using standard to parse output
Re-running tests: test attempt 2

Tests failed but no specs were found. All specs will be run again.

[14:54:27] I/launcher - Running 1 instances of WebDriver
[14:54:38] I/testLogger -
------------------------------------

[14:54:38] I/testLogger - [chrome #01-0] PID: 8196
[chrome #01-0] Specs: C:\Users\wilson.palkuri\Downloads\protractor-example-master\protractor-example-master\specs\friendSpec.js
[chrome #01-0]
[chrome #01-0] [14:54:31] I/direct - Using ChromeDriver directly...
[chrome #01-0] [14:54:38] I/runnerCli - Cannot find module 'chance'

[14:54:38] I/testLogger -

[14:54:38] E/launcher - Runner process exited unexpectedly with error code: 1
[14:54:38] I/launcher - 1 instance(s) of WebDriver still running
.[14:54:49] I/testLogger -
------------------------------------

[14:54:49] I/testLogger - [chrome #01-1] PID: 13060
[chrome #01-1] Specs: C:\Users\wilson.palkuri\Downloads\protractor-example-master\protractor-example-master\specs\nonAngularLoginSpec.js
[chrome #01-1]
[chrome #01-1] [14:54:42] I/direct - Using ChromeDriver directly...
[chrome #01-1] Jasmine started
[chrome #01-1] No Angular PASS
[chrome #01-1]
[chrome #01-1]   non-angular login
[chrome #01-1]     √ should goto friend pages on successful login
[chrome #01-1]
[chrome #01-1] Executed 1 of 1 spec SUCCESS in 1 sec.

[14:54:49] I/testLogger -

[14:54:50] I/launcher - 1 instance(s) of WebDriver still running
..[14:55:18] I/testLogger -
------------------------------------

[14:55:18] I/testLogger - [chrome #01-2] PID: 9324
[chrome #01-2] Specs: C:\Users\wilson.palkuri\Downloads\protractor-example-master\protractor-example-master\specs\qsSpec.js
[chrome #01-2]
[chrome #01-2] [14:54:54] I/direct - Using ChromeDriver directly...
[chrome #01-2] Jasmine started
[chrome #01-2]
[chrome #01-2]   Quality Shepherd blog
[chrome #01-2]     √ should display 5 posts per page
[chrome #01-2]     √ should return search results
[chrome #01-2]
[chrome #01-2] Executed 2 of 2 specs SUCCESS in 18 secs.

[14:55:18] I/testLogger -

[14:55:18] I/launcher - 0 instance(s) of WebDriver still running
[14:55:18] I/launcher - chrome #01-0 failed with exit code: 1
[14:55:18] I/launcher - chrome #01-1 passed
[14:55:18] I/launcher - chrome #01-2 passed
[14:55:18] I/launcher - overall: 1 process(es) failed to complete
[14:55:18] E/launcher - Process exited with error code 100

C:\Users\wilson.palkuri\Downloads\protractor-example-master\protractor-example-master>

wswebcreation · 2017-08-10T19:42:07Z

Hi @wpalkuri

Can you provide more info about the parser you use and the config you use?

When I look at this part of your log

[14:53:45] I/testLogger - [chrome #01-0] PID: 11920
[chrome #01-0] Specs: C:\Users\wilson.palkuri\Downloads\protractor-example-master\protractor-example-master\specs\friendSpec.js
[chrome #01-0]
[chrome #01-0] [14:53:38] I/direct - Using ChromeDriver directly...
[chrome #01-0] [14:53:45] I/runnerCli - Cannot find module 'chance'

[14:53:45] I/testLogger -

[14:53:45] E/launcher - Runner process exited unexpectedly with error code: 1
[14:53:46] I/launcher - 1 instance(s) of WebDriver still running
.[14:53:56] I/testLogger -

I see a problem in your code you don't want to rerun, but just fix ;-)

The reason why the parser (don't exactly know which one you use) is that no exception is thrown that could be picked up by the parser to determine which spec file to rerun. Are you using SpecReporter or an other tool to do the reporting? If so, try enabling the stacktrace option

wswebcreation added the question label Apr 8, 2017

This was referenced Aug 10, 2017

protractor-flake seems to rerun ALL tests when only SOME fail #6

Closed

Exception thrown with Chrom on Ubantu VM. angular/protractor#4440

Closed

qualityshepherd mentioned this issue Aug 23, 2017

feat: add support for jasmine-spec-reporter (displayStacktrace: true) #70

Merged

NickTomlin closed this as completed in #70 Aug 24, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Protractor-flake running again the tests that have passed #54

Protractor-flake running again the tests that have passed #54

rafaelcs commented Apr 8, 2017

wswebcreation commented Apr 8, 2017

rafaelcs commented Apr 8, 2017 via email

wswebcreation commented Apr 8, 2017

rafaelcs commented Apr 17, 2017 •

edited

Loading

wswebcreation commented Apr 18, 2017

surendraJ commented Jun 20, 2017 •

edited

Loading

wswebcreation commented Jun 20, 2017

surendraJ commented Jun 20, 2017

wswebcreation commented Jun 20, 2017

surendraJ commented Jun 20, 2017

wswebcreation commented Jun 20, 2017

surendraJ commented Jun 20, 2017 •

edited

Loading

wswebcreation commented Jun 20, 2017

wpalkuri commented Aug 1, 2017 •

edited by wswebcreation

Loading

wswebcreation commented Aug 10, 2017

Protractor-flake running again the tests that have passed #54

Protractor-flake running again the tests that have passed #54

Comments

rafaelcs commented Apr 8, 2017

wswebcreation commented Apr 8, 2017

rafaelcs commented Apr 8, 2017 via email

wswebcreation commented Apr 8, 2017

rafaelcs commented Apr 17, 2017 • edited Loading

wswebcreation commented Apr 18, 2017

surendraJ commented Jun 20, 2017 • edited Loading

wswebcreation commented Jun 20, 2017

surendraJ commented Jun 20, 2017

wswebcreation commented Jun 20, 2017

surendraJ commented Jun 20, 2017

wswebcreation commented Jun 20, 2017

surendraJ commented Jun 20, 2017 • edited Loading

wswebcreation commented Jun 20, 2017

wpalkuri commented Aug 1, 2017 • edited by wswebcreation Loading

wswebcreation commented Aug 10, 2017

rafaelcs commented Apr 17, 2017 •

edited

Loading

surendraJ commented Jun 20, 2017 •

edited

Loading

surendraJ commented Jun 20, 2017 •

edited

Loading

wpalkuri commented Aug 1, 2017 •

edited by wswebcreation

Loading