add a full suite of automated tests

[leob]

Right now we have 3 unit tests which are included mainly for demo purposes. It would be good if the quickstarter app has a full automated regression test suite (unit tests with Karma/Jasmine, and integration testing with Protractor and/or Appium etc).

And have the regression suite executed automatically (through travis-ci.com).

[tjaffri]

Do you mind if I take a stab at this and issue a pull request? I'm pretty much a beginner at angular but I've set up protractor e2e teats before on another project. My goal is to wore up protractor, a couple of basic e2e tests, and set up travis-ci for this repo.

[leob]

Oh cool, yes sure! Please have a shot at it, it would be nice to have some decent tests.
It doesn't have to be perfect, it's just good if we have something.

I'm beginning to get some experience with Ionic (and to a lesser degree Angular) but I'm a total beginner/newbie at the testing stuff (Jasmine, Protractor etc).

To be honest I've been postponing the testing stuff even though I know how important it is, so any contribution is more than welcome.

To give an example, I just found and fixed some really obscure bug which I've been scratching my head over for an hour (see latest commit, 9cd5f24).

The cause of this could be anything, I recently upgraded the project to Ionic 1.1.0 and AngularJS 1.4 so maybe it's related to that, who knows. This only underlines the importance of having regression tests, because now you don't know that something is broken until you happen to see it.

Yeah and travis-ci is really nice too of course.

[leob]

P.S. by the way, what would be fantastic is if you could run the e2e test suite against different builds/environments, namely:

1- the 'development' build (built with "gulp default" and served with "ionic serve" from the "src" folder)
2- the 'production' build (built with "gulp build" and then served with "ionic serve" but from the 'www' folder)
3- the 'production' build (built with "gulp build") but then when run on a real device and tested with Protractor and Appium (http://appium.io, this is a Selenium tool for iOS and Android devices)

This would give you maximum confidence that what you code and test in a development environment also works correctly with a production build, and on a device.

Each of the 3 test modes has its own advantages:

Tests running in a dev environment (1) run very fast because, by default, services are "mocked" (stubbed) to serve fake data from memory.

Tests running with a production build (2) give you assurance that minification and "strict DI" (ng-annotate) didn't mess up anything (which can easily happen of you make even a tiny mistake), and also that it works with "real" services (executing against APIs and databases and such).

And tests run with a production build on a real device (3), finally, run slowest but can be used to find device specific issues (e.g. iOS or Android specific anomalies) and can also exercise device hardware features which you can't test in mode (1) or (2).

However, if we have a basic e2e suite set up for mode (1) first, then I am reasonably sure that getting it to run in mode (2) and (3) is just a matter of configuration, and maybe some minor code tweaks.

[tjaffri]

Sounds great. My first goal is to get it running in a very basic form against "ionic serve". Then we can tune it a bit and I might need your help to get it to work against the different flavors. Yesterday I spent some time on this but ran into issues with my selenium setup. Today I'm starting from scratch in a different machine and will let you know how it goes.

[leob]

Okay cool, no worries. Take your time, there's no hurry. When you've got something then we can look at the different environments.

[tjaffri]

Created a pull request. Please let me know what you think.

[leob]

Thanks! I'll have a look at it today.

[leob]

Okay, the PR has been merged so we have the first basic e2e tests in place. Love to see them running, works like a charm.

Good start.

Later on we can add some more tests (especially if we add some more features/functionality to the starter) and we can have a look at running the e2e tests in the said 'environments'.

[leob]

Next steps: upgrade Protractor to version 3 (requires nodejs 4.x rather than 0.10.x/0.12.x), improve code structure of the e2e tests using Page Objects, Promises and Custom Matchers, add the ability to test Firebase implementations in addition to mock impls, etcetera.

[argyleink]

I've been trying to run the e2e tests this morning, is it because i'm on a newer version of node that these are failing/protractor is using an older version at the moment? it does look like an error failing at promises, and you mention above that updating to latest node and protractor is a goal. although in package.json, looks like protractor 3 is the target package version. i'm confused, cant seem to get protractor to test as expected.

[leob]

Looks fairly cryptic, the stacktrace doesn't really give any pointers about what the real problem is, I also don't see any reference to the actual e2e test code.

I see things going wrong in Selenium. I found out that Protractor has a "direct mode" which bypasses much of Selenium and is much faster. I've documented it here:

https://github.com/leob/ionic-quickstarter/wiki/Unit-testing-and-E2E-Testing#alternative-ways-to-run-protractor

Can you give this a try?

It's a matter of one simple change in protractor.conf.js and using a different command to start Protractor.

I've also added a bunch of other documentation pages to the Wiki about Protractor, unit testing and setting up Firebase and so on.

By the way, I'm using node.js version 4.2.6 and Protractor 3 but I expect it to work with newer versions too.

[argyleink]

I do plan on trying again, haven't gotten around to it yet

[leob]

Okay