Skip to content
This repository has been archived by the owner on Oct 25, 2023. It is now read-only.

add custom locator strategy #268

Merged
merged 4 commits into from
Oct 12, 2018
Merged

add custom locator strategy #268

merged 4 commits into from
Oct 12, 2018

Conversation

jlipps
Copy link
Member

@jlipps jlipps commented Oct 2, 2018

The idea behind this PR is to open up the possibility of "element finding plugins", which are third-party modules that users can use for element finding functionality that probably shouldn't be maintained by the core project but which nonetheless might be useful. (For example, I'm working on an experimental locator strategy involving a machine-learning model, which probably shouldn't become part of Appium core, at least not for some time).

The idea for how this should be used is as follows:

  1. A third party creates a node module that exports a find method matching the signature described in this PR. Essentially, that third party module is passed a driver object so it can mix and match Appium capabilities with its own logic.
  2. The user takes advantage of this plugin by setting the customFindModule capability to the name of the node module.

Right now, we assume that the node module is installed in Appium's tree (or can otherwise be required). In the future, with the Appium 2.0 architecture, we could allow plugins like these to be managed via the Appium CLI.

We might have other kinds of plugins in the future, beyond just element finding ones, but this is the first need I've come across.

Anyway, have a look and once everyone is a fan of the idea I'll work on tests etc.

@jlipps
Copy link
Member Author

jlipps commented Oct 2, 2018

An alternative approach I thought about was to encode both the module name and the selector into the custom find selector. So with the code as it stands, there are two things you need to do to make this work:

  1. Set the customFindModule cap to the name of your module (say my-find-module)
  2. Call findElement('-custom', 'mySelector')

This has the disadvantages of requiring a cap in addition to the element call, and allowing only one find plugin to be in use in a given session.

Alternatively, we could come up with a scheme like:

driver.findElement('-custom', 'my-find-module:mySelector')

Where we encode both the module and the selector into the selector field. This does away with the disadvantages above, but means that every find will be much more verbose and require client-side abstraction to manage.

I'm on the fence about which is better. What do you all think?

* @returns {WebElement} - WebDriver element or list of elements
*/
commands.findByCustom = async function (selector, multiple) {
if (!this.opts.customFindModule) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we allow only one such module to be set at the same time or many of them?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the way the code is written now, just one module. but see the discussion we're having about whether we want something else.

*/
commands.findByCustom = async function (selector, multiple) {
if (!this.opts.customFindModule) {
throw new Error("Finding an element using a pluagin is currently an " +
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo: plugin

if (!this.opts.customFindModule) {
throw new Error("Finding an element using a pluagin is currently an " +
"incubating feature. To use it you must manually install a " +
"plugin module inside your Appium node_modules tree, " +
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we explicitly say where to install it? npm might change this in the future though

throw new Error(`Could not load your custom find module. Original error: ${err}`);
}

if (!_.isFunction(finder.find)) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

finder && !_.isFunction ... would be safer

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, i guess it's possible that a module is not truthy?

try {
finder = require(this.opts.customFindModule);
} catch (err) {
throw new Error(`Could not load your custom find module. Original error: ${err}`);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd put the actual module name instead


let finder;
try {
finder = require(this.opts.customFindModule);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this secure in case one adds some malicious module? do we want to check some extra stuff here?

Copy link
Member Author

@jlipps jlipps Oct 2, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'm not sure what else we can check. it's the users responsibility to vet 3rd-party plugins, which could indeed be malicious.

probably at some point we'll have a registry of 'approved' plugins that we have vetted.


const customFinderLog = logger.getLogger(this.opts.customFindModule);

let els;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd rather change it to elements. The current one is not very readable


// if we're looking for multiple elements, or if we're looking for only
// one and found it, we're done
if (els.length || multiple) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

!_.isEmpty() is easier to read

throw err;
}

if (multiple) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

return multiple ? els : els[0]

@KazuCocoa
Copy link
Member

This has the disadvantages of requiring a cap in addition to the element call, and allowing only one find plugin to be in use in a given session.

Can we relax the condition by adding the caps as an array like ['my-find-module1', 'my-find-module2']? But in the case, mySelector in Call findElement('-custom', 'mySelector') should be unique in all included modules to avoid conflict each selector name..
This way is implicit. If a user specifies many custom modules, understanding which selector belongs to which module in all modules is probably difficult and complicated.

Personally, I like below way to make managing them easier. (More explicit way)

driver.findElement('-custom', 'my-find-module:mySelector')


If we'll allow only one custom module against a session, the original way, driver.findElement('-custom', 'my-find-module:mySelector'), is better though.

@dpgraham
Copy link
Contributor

dpgraham commented Oct 2, 2018

I like the idea but I'm concerned about the practice of inserting a node module into Appium's existing node_modules because that may require tampering with versions of packages and it may break the determinism of the shrinkwrap.

May I suggest that the plugins be standalone node packages that sit somewhere on the filesystem, either in some permanent directory that stores plugins globally or in a plugins directory that's in the same directory as Appium (ignored by npm and git). The plugins can be required like require('/path/to/finder-module.js') and dependencies are injected into the package.

Something like:

// magic-finder-plugin.js
function pluginInitializer(appiumDepOne, appiumDepTwo, appiumDepThree, ...) {
   ...
}

export {pluginInitializer};

// finder-plugins.json
[
  {name: 'magic-finder-plugin', path: '/path/to/magic-finder-plugin.js'}
]

// appium-base-driver.js
function initializeFinderPlugins() {
  const plugins = require('./plugins.json');
  for (let plugin of plugins) {
    const {pluginInitializer} = require(plugin.path);
    finderPlugin(appiumDepOne, appiumDepTwo, appiumDepThree, ...);
  }
}

@dpgraham
Copy link
Contributor

dpgraham commented Oct 2, 2018

What if we made the plugins a server-flag and when you start the server it installs the plugins (if they're not already there)

@jlipps
Copy link
Member Author

jlipps commented Oct 2, 2018

Thanks for the thoughts so far @mykola-mokhnach @KazuCocoa @dpgraham. There are now two design questions on the table, each with a few proposed answers:

  1. How do we provide access to custom find functionality?
    a. Through a customFindModule cap in conjunction with -custom loc strat
    b. Through a -custom loc strat in conjunction with a complex selector of the form <module>:selector
  2. How do we support the finding and loading of plugins?
    a. We do not support it. We simply require the named module and force the user to make it require-able by Appium (whether by putting it into Appium's tree, or making it globally available)
    b. Have the user give us two pieces of info about the module: its name and its location. This would allow us to require it without it being in Appium's tree. The user is still in charge of installing it
    c. Have the user give us the name and version spec of the module, and we put it somewhere and manage it for them.

As for design question (1), it seems like the consensus so far is that (1a) would be potentially too restrictive, and a more flexible approach would be better so that we could choose from a number of plugins. Are there any other proposals than (1b)?

As for design question (2), the proposals raise some interesting considerations. I went initially with (2a) because anything else encroaches on the territory taken up by Appium 2.0. I wanted to be careful not to design something that we would just throw away when we finally decide on an implementation for Appium 2.0.

Alternatively, we could use this as an opportunity to actually move forward with the Appium 2.0 design in an area which isn't super crucial, namely this new plugin feature. We would just need to put the time into making good decisions. At a first pass, something like the appium-drivers.json in the Appium 2 proposal makes sense, only expanding it to account for both "drivers" and "plugins" (maybe in the first instance we just worry about plugins, and then later port driver management to the same system).

As for your specific suggestion, @dpgraham:

What if we made the plugins a server-flag and when you start the server it installs the plugins (if they're not already there)

I like this suggestion a lot, however if we are moving to a CLI installer model for Appium 2.0 (appium driver install xcuitest), then I would want to do the same thing for plugins (appium plugin install custom-element-finder@1.2.0). (Of course, there's no reason we couldn't in addition have a server flag that auto-installs modules on server start: appium --plugins=custom-element-finder@1.2.0)

I'm open here; just wanted to start out with something simple that advanced users could play around with while we determined whether this whole plugin thing was truly useful. Thoughts? Would love to hear @imurchie's views as well.

@dpgraham
Copy link
Contributor

dpgraham commented Oct 11, 2018

Lgtm

@jlipps
Copy link
Member Author

jlipps commented Oct 11, 2018

@dpgraham @KazuCocoa @mykola-mokhnach just pushed a change that will allow users to register multiple element finder plugins as an object, allowing for a 'shortcut' name. for example, i could register one like this:

{"customFindModules": {"1": "/really/long/path/to/find/plugin/module/"}}

and then in my test code:

driver.findElement('-custom', '1:selector');

so I can just use 1 as my local alias for that particular plugin. (we also handle the case where there is only one plugin registered, in which case a prefix is not required).

take a look at the code, and i'll start working on unit tests.

@jlipps
Copy link
Member Author

jlipps commented Oct 11, 2018

tests pushed

Copy link
Member

@KazuCocoa KazuCocoa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me 👍

it('should be able to use multiple find modules', async function () {
const d = new BaseDriver();
d.opts.customFindModules = {f: CUSTOM_FIND_MODULE, g: CUSTOM_FIND_MODULE};
await d.findElement(CUSTOM_STRATEGY, "f:foo").should.eventually.eql("bar");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@jlipps jlipps merged commit 02b3b87 into master Oct 12, 2018
@jlipps jlipps deleted the jlipps-custom-loc-strat branch October 12, 2018 17:20
@jlipps
Copy link
Member Author

jlipps commented Oct 12, 2018

thanks everyone! published as 3.9.0.

once appium 2 rolls around we can merge this into whatever the ultimate plugin structure becomes.

@MurrSch
Copy link

MurrSch commented Oct 19, 2018

Hi @jlipps, is adding the custom locator strategy in the ruby-client on the horizon? :)

@KazuCocoa
Copy link
Member

let's try out the latest ruby_lib_core (2.0.4)
appium/ruby_lib_core#151

@SoundaryaBhavanam
Copy link

I am able to launch the app with the below capabilities
HashMap<String, String> customFindModules = new HashMap<>();
customFindModules.put("ai", "test-ai-classifier");
caps.setCapability("customFindModules", customFindModules);
caps.setCapability("shouldUseCompactResponses", false);

But not able to find the text box with the visible text like

driver.findElement(MobileBy.custom("ai:User ID"))

getting the below exception

Element info: {Using=-custom, value=ai:User ID}
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.openqa.selenium.remote.ErrorHandler.createThrowable(ErrorHandler.java:214)
at org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:166)
at org.openqa.selenium.remote.http.JsonHttpResponseCodec.reconstructValue(JsonHttpResponseCodec.java:40)
at org.openqa.selenium.remote.http.AbstractHttpResponseCodec.decode(AbstractHttpResponseCodec.java:80)
at org.openqa.selenium.remote.http.AbstractHttpResponseCodec.decode(AbstractHttpResponseCodec.java:44)
at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:158)
at io.appium.java_client.remote.AppiumCommandExecutor.execute(AppiumCommandExecutor.java:239)
at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:548)
at io.appium.java_client.DefaultGenericMobileDriver.execute(DefaultGenericMobileDriver.java:42)
at io.appium.java_client.AppiumDriver.execute(AppiumDriver.java:1)
at io.appium.java_client.ios.IOSDriver.execute(IOSDriver.java:1)
at org.openqa.selenium.remote.RemoteWebDriver.findElement(RemoteWebDriver.java:322)
at io.appium.java_client.DefaultGenericMobileDriver.findElement(DefaultGenericMobileDriver.java:62)
at io.appium.java_client.AppiumDriver.findElement(AppiumDriver.java:1)
at io.appium.java_client.ios.IOSDriver.findElement(IOSDriver.java:1)
at io.appium.java_client.FindsByCustom.findElementByCustom(FindsByCustom.java:38)
at io.appium.java_client.MobileBy$ByCustom.findElement(MobileBy.java:628)
at org.openqa.selenium.remote.RemoteWebDriver.findElement(RemoteWebDriver.java:314)
at io.appium.java_client.DefaultGenericMobileDriver.findElement(DefaultGenericMobileDriver.java:58)
at io.appium.java_client.AppiumDriver.findElement(AppiumDriver.java:1)
at io.appium.java_client.ios.IOSDriver.findElement(IOSDriver.java:1)
at Testing.AI.appiumclassifierplugin.AppTest.testFindElementUsingAI(AppTest.java:90)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:580)
at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:716)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:988)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
at org.testng.TestRunner.privateRun(TestRunner.java:648)
at org.testng.TestRunner.run(TestRunner.java:505)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:455)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:450)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:415)
at org.testng.SuiteRunner.run(SuiteRunner.java:364)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:84)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1208)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1137)
at org.testng.TestNG.runSuites(TestNG.java:1049)
at org.testng.TestNG.run(TestNG.java:1017)
at org.testng.remote.AbstractRemoteTestNG.run(AbstractRemoteTestNG.java:114)
at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:251)
at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:77)
Can some one help me to find the elements ?

@KazuCocoa
Copy link
Member

@SoundaryaBhavanam
Could you create the issue in https://github.com/testdotai/appium-classifier-plugin if your question is the AI logic?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants