Discussing a standard for cross-platform UI-based app automation

[jlipps]

Hi everyone, I'm a maintainer of the Appium project (part of the OpenJS Foundation). For some time I've been thinking about an argument/proposal to do with an open standard for UI-based app automation. Appium emerged almost a decade ago as a sort of parallel to Selenium--Selenium focused on browser automation, and Appium focused on mobile. Nowadays Appium sees itself as a platform for developing automation support for any platform, not just mobile (TV, IoT, gaming consoles, car screens, etc...)

Appium chose to rely on Selenium's protocol, which became the official W3C WebDriver standard, and it still keeps in step with that standard (for example, many of Appium's client libraries are built on top of Selenium clients which implement the WebDriver protocol). As Appium's purview kept expanding, it became necessary to extend the WebDriver protocol with new behaviours (since platforms other than the web have different automation requirements). Given the success of Appium in terms of making the WebDriver protocol one of the more popular ways to automate apps on all kinds of platforms, I think the time has come to discuss standardizing the protocol work Appium has done.

There are a number of motivations for standardization, but the main one is to convince and attract app platform vendors to create and maintain their own implementations of the "Appium" standard, which have a better chance of being high quality and in sync with their own internal code. This is the transition that happened with the advent of the WebDriver protocol, where Safari, Google, Edge, Firefox, etc..., all began to ship with their own WebDriver implementations. The implementation burden shifted from the Selenium project to the browser vendors themselves, where it arguably belongs. I think there are many good reasons the same story can and should happen with app platforms beyond the browser.

I'd welcome some of your time to present a brief version of this argument and get your feedback!

[jorydotcom]

@christian-bromann @boneskull - in case you can make the meeting next week!

[jlipps]

OpenJS standards - Standards proposal.pdf
Attaching the presentation I used at this meeting for reference

[michaelchampion]

"Let's make a standard to pressure platform vendors to support it" seldom works in my experience (on the browser side anyway). Don't start with a solution (make a standard) and justify it, explain the problem and lead the audience to the conclusion that an Appium-based standard would help solve it.

As best I understand from the presentation, the problem is that open source automation tools such as Appium can't do as good a job as vendor-written tools that can access low level internal APIs. But vendor-written tools frustrate developers/tests who work on multiple platforms.. It also raises the bar for new platform entrants, who not only have to develop the platform itself but the test automation tools. (Vendors care about regulators, who care about platform lock-in / barriers to entry).

Is that about right?

If so, I suspect you need to be more specific about what specific functions that Appium depends on (or a future Appium would be improved if they exists) could be common across platforms to address the problem. To the extent they could be prototyped / demo'd on an open source app platform (does a viable one exist these days?) that would help the case.

Once vendors are convinced that common lower-functionality to facilitate UI test automation is needed, THEN the question of "how do we work together to make that happen" becomes relevant. A standards effort is the obvious answer if the need is for SPECs rather than open source code. The question of where to do the work becomes completely tactical -- is an existing SDO such W3C or ECMA fit for this purpose? Would a special-purpose organization under the LF Joint Development Foundation be more efficient for this job?

[jlipps]

@michaelchampion thanks for reviewing!

explain the problem and lead the audience to the conclusion that an Appium-based standard would help solve it.

That's what I'm trying to do, anyway. My goal is to first work with the vendors to get them to see the current problematic state and to see if they agree a standard can be a solution. I think there could be other, less optimal, solutions that don't involve standards at all, that could also benefit the community.

You're absolutely right that the question of a venue for standards development is kind of irrelevant at this stage and is a tactical question for later.

Is that about right?

Yes, you got it. I think there is an argument (which I've spelled out more fully in a manifesto of sorts I decided not to inflict this group with) for why a lingua franca for UI automation is beneficial for vendors, developers, and users. My goal right now is to (a) make sure that's a good argument, and then (b) share it with people at the important vendors who have the authority and capacity to evaluate it. Otherwise, as you pointed out, there would be no buy-in from the relevant parties for a standard and the whole thing is moot.

If so, I suspect you need to be more specific about what specific functions that Appium depends on

The functionality described in the WebDriver standard is a good starting point. The functionality available via existing UI automation tools (e.g., XCUITest and Espresso) are great lists of functionality as well. The problems Appium has include:

• the functionality available differs wildly by platform
• not all functionality is publicly available (Appium relies in some cases on private frameworks/headers)
• Appium has to make reactive changes to platform updates
• the semantics of behaviours sometimes differs from platform to platform (e.g., a "scroll" on iOS may not be the same as a "scroll" on Android when using the vendor-provided automation frameworks)

At the end of the day, even if all a vendor did was take on the responsibility of maintaining a webdriver-compatible interface to their proprietary automation technology, that would be a huge win for Appium and devs trying to test on the platform. This is something like what Microsoft has already done with WinAppDriver (which was much beloved by the community, though sadly they've pulled resources from the project and it's languished in recent years).

So in that sense, vendors don't even necessarily need to agree on common lower-level functionality or equivalent semantics for their automation behaviours, though that would obviously be an ideal scenario. Even just making a best-effort attempt at maintaining their own interface for a standard protocol (I'd argue for an extension of the webdriver protocol given its status) would be a huge step forward.