Add screenshot comparison capability

[jkeys089]

The purpose of this issue is to start a conversation about wether or not you might be interested in a contribution of a new UI testing capability: screenshot comparison.

We've built a screenshot comparison feature that we've been using internally with good results for a while. We actually started with a commercial service (https://percy.io/) and then migrated to an OSS project running locally (https://github.com/Mikuu/Micoo) for a bit more control. After running into a few issues we realized we needed something integrated with Karate and we built one.

Some of the issues we encountered using external screenshot comparison services include:

• Difficulty sharing a single set of baseline images across feature branches (e.g. developers working on different features will run into failures until their updates can be included in the set of baseline images)
• Hosted services are limiting for remote developers (e.g. it isn't possible to run screenshot comparisons without a fast, reliable internet connection)
• Hosted services have inherent limitations (e.g. limited number of screenshots with costly overage penalties when using commercial services or single-threaded performance in Micoo)

Our ideal solution:

• Run screenshot comparisons in realtime as we take them (even when running multi-threaded tests)
• Define comparison settings inline with the tests where the screenshots are taken
• Review comparison results and modify screenshot settings directly in the Karate reports
• Check-in baseline screenshots and comparison configs with the tests (e.g. so developers could make updates in feature branches independent of other branches)

We accomplished our goals by:

• Adding a diffImg function that accepts a named image and a baseline screenshot directory along with an optional configuration
• Adding pure Java image comparison routines based on Resemble.js and ssim.js
• Adding a diff UI based on Resemble.js and also inspired by Micoo that is embedded into Karate reports via doc to easily review comparison results or update screenshot configurations (e.g. to ignore dynamic areas of screenshots such as animations)

My question is: would you be interested in accepting this feature into the main Karate project or would it be better to keep this as a separate project?

If you're interested, we'd be happy to contribute the feature and collaborate with the Karate team to ensure it fits naturally.

[ptrthomas]

@jkeys089 we are very open to the idea. if I understand you right, the pure Java image comparison routines are written from scratch, which sounds really good.

one question is how much does this add to the maven dependencies. depending on that - we can take a call if this has to be part of the main karate project, or even within karate-core itself.

really appreciate your work on this.

[jkeys089]

@ptrthomas great! Yes, the image comparison code is ported from Javascript to Java by hand and adds no new dependencies. It may make sense to split those out as separate projects and then pull them in as dependencies. For the report UI we added two additional dependencies: Resemble.js and jQuery UI (used for dragging / resizing ignored boxes)

It will take us a bit of work to clean everything up and remove bits that are specific to our needs. I guess the initial PR should be ready in a week or two once we get started.

[ptrthomas]

@jkeys089 looking forward to it ! the JS dependencies sound good to me, we currently bundle a few including jQuery

[ptrthomas]

1.3.0 released