Update e2e test dependency updates & get them running on Mac #2083

benjaminapetersen · 2017-09-15T04:09:27Z

Based off of #2068, though not really dependent. Related to #1473, a previous attempt at getting our e2e tests running.

Depends on #2268 for junit additions.

NOTE: this PR currently contains a backup of node_modules/protractor in test/package_backups/protractor, which accounts for many of the listed files changed (166!)

TODO:

Update dependencies
Update .conf.js files
Eliminate special Mac conf if possible
Create helpers for testing non-angular pages (login page) - helpers/nonAngular.js
Get e2e tests running locally in Chrome
~~Get e2e tests running locally in Firefox~~
~~Get e2e tests running in PhantomJS or Nightmare~~
Get e2e tests running in CI in headless Chrome
Fix tests broken since updates

ISSUES:

~~Accepting certs does not work in Firefox~~ (now using chrome)
~~Need to run grunt serve once in order to accept certs~~ (not with chrome)
Webdriver manager is a huge hassle due to a dependence on Github's API, which is aggressively rate-limited when non-authenticated. It attempts to use the API to detect a good Selenium server to download. npm install works fine outside of the office, but in office the rate-limit is always exceeded (IP address specific I suppose). This is a known issue, links in comments below. (appears to have resolved on its own, unfortunately I think these dependencies can change under us)

Firefox ESR download links for testing locally (match jenkins):

FF 52.4 (latest & broken)
FF 52.3 (latest working for our tests)
(this PR/branch will not work with these verisons of FF, but our current setup with Protractor 1.x should work)

For reference, Downgrade of FF in Jenkins PR
For reference, jenkins config file

@spadgett

spadgett · 2017-09-15T11:33:34Z

@benjaminapetersen 👍 nice work

benjaminapetersen · 2017-09-15T16:01:04Z

@stevekuznetsov I may request some assistance from you here as well soon. Working on the other PR I pinged you on first as unit tests should be easier than e2e to get up and running again...

benjaminapetersen · 2017-09-18T18:15:06Z

Been working on getting some progress with webdriver-manager hitting rate limit issues with github & not being able to update.

angular/webdriver-manager#270
angular/webdriver-manager#269

I don't see much activity in webdriver-manager since May, not sure if that bodes well.

benjaminapetersen · 2017-10-03T14:02:15Z

[test]

benjaminapetersen · 2017-10-03T14:29:35Z

[test]

benjaminapetersen · 2017-10-03T15:08:15Z

Sigh. Welp, twas' working a week ago. Wheee!

Tempted to suggest converting e2e tests to unit tests & dropping the whole e2e flow entirely....

benjaminapetersen · 2017-10-03T15:26:43Z

Issue in protractor tracking one of our problems. Our chrome-driver dependency is not latest, so we are getting a bug (see the convo), however, karma-chrome-launcher is latest, so I'm not exactly sure how to nail down all the dependencies down the tree that keep giving us headaches.

second issue where protractor is incorrectly seeing both capabilities & multiCapabilities.

benjaminapetersen · 2017-10-03T15:45:33Z

issue in webdriver manager tracking one of our problems.
another issue in webdriver manager tracking one of our problems.

benjaminapetersen · 2017-10-03T19:54:03Z

[test] plz

benjaminapetersen · 2017-10-03T20:43:26Z

Issues:

visiting url: create-project
[20:37:02] E/protractor - Could not find Angular on page https://localhost:9000/dev-console/create-project : retries looking for angular exceeded
Fskipping window.setSize()...
visiting url: create-project
[20:37:13] E/protractor - Could not find Angular on page https://localhost:9000/dev-console/create-project : retries looking for angular exceeded
Fskipping window.setSize()...
visiting url: create-project
[20:37:23] E/protractor - Could not find Angular on page https://localhost:9000/dev-console/create-project : retries looking for angular exceeded
FF
...
   Failed: Angular could not be found on the page https://localhost:9000/dev-console/create-project.If this is not an Angular application, you may need to turn off waiting for Angular.
                              Please see 
                              https://github.com/angular/protractor/blob/master/docs/timeouts.md#waiting-for-angular-on-page-load
  Stack:
    Error: Angular could not be found on the page https://localhost:9000/dev-console/create-project.If this is not an Angular application, you may need to turn off waiting for Angular.
...

🙁

benjaminapetersen · 2017-10-03T21:21:11Z

lolz... welp, runs locally, won't run in travis. flipped the problem around now. 🙄

benjaminapetersen · 2017-10-04T02:30:38Z

[test]

benjaminapetersen · 2017-10-04T03:02:20Z

[test]

spadgett · 2017-10-04T13:08:55Z

@benjaminapetersen I think you need to get screenshots working again to be able to debug in Jenkins.

spadgett · 2017-10-04T16:59:39Z

[test]

benjaminapetersen · 2017-10-04T17:40:37Z

[test]

benjaminapetersen · 2017-10-04T18:51:08Z

yay screenshots

benjaminapetersen · 2017-10-17T20:24:52Z

/retest

benjaminapetersen · 2017-10-18T13:34:00Z

Attempting to work out the flakes running the tests in CI. Opened an issue with Hawtio as the manual bootstrapping (on the document) may be causing some of the issue:

Hawtio 2398 and manual bootstrap instructions.

Also, angular.bootstrap function docs page states:

Note: Protractor based end-to-end tests cannot use this function to bootstrap manually. They must use ngApp.

benjaminapetersen · 2017-10-18T16:14:58Z

/test

dtaylor113 · 2017-10-25T15:46:08Z

Hi, just cloned this PR branch and after upgrading my Chrome browser (linux/Fedora v62), e2e tests passed.

benjaminapetersen · 2017-10-25T15:49:08Z

Great, thanks @dtaylor113.

So far then, @benjaminapetersen, @dtaylor113 & @rhamilto have run successfully, but @spadgett has not been able to get them to run. CI is currently flaky, but I'm pretty sure I know why & will try to address that today if possible.

benjaminapetersen · 2017-10-25T18:32:17Z

/test

benjaminapetersen · 2017-10-25T19:17:03Z

Jenkins / Travis success.

/test again (going for 3x)

benjaminapetersen · 2017-10-25T19:50:09Z

@stevekuznetsov 😠 😡 😠 😡
Just cuz I'm sure you want to know about all my rage faces.

benjaminapetersen · 2017-10-25T19:50:18Z

/retest

- Upgrade protractor & dependencies - Rewrite tests to match new APIs - Remove deprecated tests - Remove protractor-mac.conf.js & use a single protractor config - Add `user logs in` spec to work out bugs with the login flow. - Fix login flow (non-angular page), update some helpers to new protractor apis, fixing individual tests in progress - Add temp e2e grunt task to make testing quicker - Add additional scrollTo window helpers - Update createFromTemplate page to eliminate test flakes due to protractor changes - Add nextSteps page pageObject - Remove chrome from multiCapabilities in protractor.conf so grunt can handle setting the browser - Update framework from jasmine to jasmine2 in protractor.conf.js - Deduplicate common setup across test suites - Add implicit browser.wait(1000) every time new PageObject() or PageObject.visit() to ensure items are properly rendered before tests continue - Update support for screenshots. Screenshots will be saved in /test/tmp/<browser>/<spec_title>.png - Screenshots taken for failed tests only - Add new user_creates_project e2e test that exercises our menus (rather than browser page refreshes) - Always default protractor to run with Chrome - Standardize timing across modules - Improve reporting output of protractor tests - Use color output locally on mac for test output, plain text in Jenkins - Jenkins does not do well with the escape codes - Prow (Jenkins) should now use the JUnit xml files to give us better context on failure anyway

…our flow

benjaminapetersen · 2017-12-20T17:10:28Z

rebased, happily was not painful.

…ization

openshift-ci-robot · 2018-01-02T16:41:26Z

@benjaminapetersen: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
ci/openshift-jenkins	`16dc281`	link	`/test`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-merge-robot · 2018-01-26T08:01:08Z

@benjaminapetersen PR needs rebase

benjaminapetersen · 2018-02-08T03:18:34Z

Revisited this, and got over the Hawtio hump (I think) but we will now hit this:

Error: Cannot find module '_debugger'
    at Function.Module._resolveFilename (module.js:555:15)
    at Function.Module._load (module.js:482:25)
    at Module.require (module.js:604:17)
    at require (internal/module.js:11:18)

angular/protractor#4307

Doesn't look like the protractor team is going to address that anytime soon. Its still usable, but its a node 8 bug thats been sitting for quite a while.

spadgett · 2018-02-08T12:29:04Z

I think we'd see that node 8 error without these changes. That seems to just be a protractor problem.

benjaminapetersen · 2018-02-08T15:00:38Z

Yup, its just general.... just a little concerned the protractor & webdriver-manager teams are not terribly responsive anymore.

spadgett mentioned this pull request Sep 16, 2017

Enable new landing page experience by default #2078

Merged

3 tasks

openshift-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 16, 2017

benjaminapetersen force-pushed the bpetersen/e2e-test-dependency-updates branch from 43e9fd2 to e553d83 Compare September 18, 2017 03:32

openshift-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 18, 2017

benjaminapetersen force-pushed the bpetersen/e2e-test-dependency-updates branch from e553d83 to 41ffaeb Compare September 19, 2017 14:16

openshift-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 23, 2017

benjaminapetersen force-pushed the bpetersen/e2e-test-dependency-updates branch 2 times, most recently from e61b45f to 2897413 Compare September 29, 2017 19:39

openshift-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 29, 2017

benjaminapetersen mentioned this pull request Oct 2, 2017

Test Flake: A Jasmine spec timed out. Resetting the WebDriver Control Flow. #2200

Closed

benjaminapetersen mentioned this pull request Oct 4, 2017

Protractor incorrectly assumes I have both capabilities & multi-capabilities angular/protractor#4520

Closed

spadgett mentioned this pull request Oct 4, 2017

Temporarily disable some integration tests to unblock queue #2210

Merged

benjaminapetersen mentioned this pull request Oct 4, 2017

Add Chrome for console integration tests openshift-eng/aos-cd-jobs#706

Merged

benjaminapetersen mentioned this pull request Oct 25, 2017

Configuring protractor against an app using HawtioPluginLoader is unclear hawtio/hawtio-core#14

Closed

benjaminapetersen mentioned this pull request Dec 7, 2017

[WIP] Remove create project and edit project pages #1954

Closed

benjaminapetersen added 8 commits December 20, 2017 12:05

f: add wait statements to matchers

ae08240

WIP: projectHelpers.deleteAllProjects() seems to be problemating for …

6eacca6

…our flow

WIP: comment out deleteAllProjects() helper

37597d3

Update beforeAll,afterAll,beforeEach,afterEach

28c2f55

Add user_deletes_all_projects.spec.js (empty)

4a0b155

Update delete projects flow

37ae78d

jshint fix

75fd47b

benjaminapetersen force-pushed the bpetersen/e2e-test-dependency-updates branch from 8d81f24 to 75fd47b Compare December 20, 2017 17:09

benjaminapetersen added 2 commits December 21, 2017 12:34

Fix login flow test component with better toggles for Angular syncron…

63742a4

…ization

Add more logging for debugging purposes

16dc281

openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 26, 2018

spadgett closed this Jan 24, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update e2e test dependency updates & get them running on Mac #2083

Update e2e test dependency updates & get them running on Mac #2083

benjaminapetersen commented Sep 15, 2017 •

edited

Loading

spadgett commented Sep 15, 2017

benjaminapetersen commented Sep 15, 2017

benjaminapetersen commented Sep 18, 2017

benjaminapetersen commented Oct 3, 2017

benjaminapetersen commented Oct 3, 2017

benjaminapetersen commented Oct 3, 2017 •

edited

Loading

benjaminapetersen commented Oct 3, 2017 •

edited

Loading

benjaminapetersen commented Oct 3, 2017

benjaminapetersen commented Oct 3, 2017

benjaminapetersen commented Oct 3, 2017

benjaminapetersen commented Oct 3, 2017

benjaminapetersen commented Oct 4, 2017

benjaminapetersen commented Oct 4, 2017

spadgett commented Oct 4, 2017

spadgett commented Oct 4, 2017

benjaminapetersen commented Oct 4, 2017

benjaminapetersen commented Oct 4, 2017

benjaminapetersen commented Oct 17, 2017

benjaminapetersen commented Oct 18, 2017 •

edited

Loading

benjaminapetersen commented Oct 18, 2017

dtaylor113 commented Oct 25, 2017

benjaminapetersen commented Oct 25, 2017

benjaminapetersen commented Oct 25, 2017

benjaminapetersen commented Oct 25, 2017

benjaminapetersen commented Oct 25, 2017

benjaminapetersen commented Oct 25, 2017

benjaminapetersen commented Dec 20, 2017 •

edited

Loading

openshift-ci-robot commented Jan 2, 2018

openshift-merge-robot commented Jan 26, 2018

benjaminapetersen commented Feb 8, 2018

spadgett commented Feb 8, 2018

benjaminapetersen commented Feb 8, 2018

Update e2e test dependency updates & get them running on Mac #2083

Update e2e test dependency updates & get them running on Mac #2083

Conversation

benjaminapetersen commented Sep 15, 2017 • edited Loading

spadgett commented Sep 15, 2017

benjaminapetersen commented Sep 15, 2017

benjaminapetersen commented Sep 18, 2017

benjaminapetersen commented Oct 3, 2017

benjaminapetersen commented Oct 3, 2017

benjaminapetersen commented Oct 3, 2017 • edited Loading

benjaminapetersen commented Oct 3, 2017 • edited Loading

benjaminapetersen commented Oct 3, 2017

benjaminapetersen commented Oct 3, 2017

benjaminapetersen commented Oct 3, 2017

benjaminapetersen commented Oct 3, 2017

benjaminapetersen commented Oct 4, 2017

benjaminapetersen commented Oct 4, 2017

spadgett commented Oct 4, 2017

spadgett commented Oct 4, 2017

benjaminapetersen commented Oct 4, 2017

benjaminapetersen commented Oct 4, 2017

benjaminapetersen commented Oct 17, 2017

benjaminapetersen commented Oct 18, 2017 • edited Loading

benjaminapetersen commented Oct 18, 2017

dtaylor113 commented Oct 25, 2017

benjaminapetersen commented Oct 25, 2017

benjaminapetersen commented Oct 25, 2017

benjaminapetersen commented Oct 25, 2017

benjaminapetersen commented Oct 25, 2017

benjaminapetersen commented Oct 25, 2017

benjaminapetersen commented Dec 20, 2017 • edited Loading

openshift-ci-robot commented Jan 2, 2018

openshift-merge-robot commented Jan 26, 2018

benjaminapetersen commented Feb 8, 2018

spadgett commented Feb 8, 2018

benjaminapetersen commented Feb 8, 2018

benjaminapetersen commented Sep 15, 2017 •

edited

Loading

benjaminapetersen commented Oct 3, 2017 •

edited

Loading

benjaminapetersen commented Oct 3, 2017 •

edited

Loading

benjaminapetersen commented Oct 18, 2017 •

edited

Loading

benjaminapetersen commented Dec 20, 2017 •

edited

Loading