Testing: CI

[fredck]

The Testing page should have a section about continuous integration. We must agree whether, how and what to have in this sense.

[fredck]

@gregpabian said:

We could consider starting using services like Travis or Coverals?

[jodator]

+1 (without arguing ;-) )

[oleq]

We could consider starting using services like Travis or Coverals?

👍

[Reinmar]

Noo... let's do this manually. Automation is for losers.

(Trying to keep our threads consistently hot.)

[gregpabian]

@Reinmar

[pjasiun]

👍 (boooooring) But we could start argue if we should use Travis or Coverals :)

[Reinmar]

I remember that we had a discussion whether single plugin's tests should be runnable from its directory. I can't find where the discussion was, but for now it seems that we decided that the tests will be run from the ckeditor5 repository.

I though that while this is pretty ok for the development process because we KISS, it makes it hard (impossible?) to have CI for your plugins. It's not a big deal for us because we can test all our plugins and modules from ckeditor5 repository, but it will be a pain for all 3rd part plugins' developers as well as for our unofficial plugins. Basically, it unfortunately should be possible to easily run tests from plugin's directory so plugins can be integrated with any CI.

[fredck]

I was wondering if it would be ever possible to bring all the necessary stuff for testing through a npm module. It would inject ckeditor5 and ckeditor5-core, as well all configurations and the such... it would be a bit of magic, but maybe it is doable.

[Reinmar]

We've discussed this today and we're not going to work on allowing running tests from subpackages directories. @jodator has worked on setting up CI on Jenkins for CKEditor 5 and it turns out that it requires too many hacks because of the

1. Dev dependencies (like Bender, its plugins, sinon, etc.) must be specified in the dependencies section of ckeditor5 package's package.json which will slow down its installation (in scenario like in How to create a CKEditor 5 setup without forking ckeditor5 repo? #112). Currently they are defined in devDependencies list so they are installed only when running npm install inside a cloned ckeditor5 repo (so during actual development).
2. When installing dependencies starting from ckeditor5-foo package we get Bender packages inside ckeditor5-foo/node_modules what breaks Bender runner. It expects that all paths (requirements, plugins, config, application, tests) are related to the place where you run bender server run. Making it accept cwd would require a lot of work.
3. ckeditor5 must be defined as a dependency of each subpackage.

Therefore we decided that CI must do the following:

1. Clone ckeditor5.
2. Remove all ckeditor5-* entries from devDependencies (because we want to test only the single one).
3. Add the one we want to test with a correct GitHub URI (pointing to the right commit).
4. Run npm install.
5. Run gulp build.
6. Run bender run -b <browsers>.

[Reinmar]

As for testing ckeditor5 I think that we can do just steps 1, 4, 5, 6, so with all the devDeps. This will be kinda like integration tests, because we plan to add all our packages to devDeps (as a default dev setup).

[jodator]

We've discussed this today and we're not going to work on allowing running tests from subpackages directories

That's sad since I think that much cleaner solution would be:

1. in each ckeditor5-foo define it's dependencies
2. move builder to separate repository like ckeditor5-builder
3. build ckeditor5 either from ckeditor5 as it is now or from each ckeditor5-* package

This would allow to run bender from ckeditor5-foo repo.

Anyway this would require to adapt builder script for this so probably not going to happen and solution described by @Reinmar would probably work out of the box.

[Reinmar]

move builder to separate repository like ckeditor5-builder

I've been also considering this option. But now the truth is that ckeditor5 contains the builder and just a couple other small things like bender.js. So:

1. The situation would be pretty similar to what we have now.
2. Where would we put bender.js if we moved Bender out of ckeditor5?

[jodator]

Where would we put bender.js if we moved Bender out of ckeditor5?

Probably separate per repo (even if some boilerplate will be needed).

Also ckeditor5 would have all existing features/plugins so it will act as integration repo to run all the test so it would also have ckeditor5-builder as it's dependency. Also still can be used as dev repo with its scripts for developing taksks. Only the builder could be reusable by others.

[Reinmar]

As for testing ckeditor5 I think that we can do just steps 1, 4, 5, 6, so with all the devDeps. This will be kinda like integration tests, because we plan to add all our packages to devDeps (as a default dev setup).

We can also have a special tag for integration tests and when running CI tests for subpackages ignore these (run them only when running tests from ckeditor5).

To clarify – each subpackage can define its integration tests. Dependencies (other subpackages) of these integration tests should be added to devDeps of this subpackage so if someone runs them without having all packages installed, they would pass.

[Reinmar]

We'll continue this work in ckeditor/ckeditor5#354.