synthetic events do not take into account overlapping elements

[lddubeau]

Versions

Selenium version: 2.46.0
Browser: FF 34
OS: I reproduce this on Linux but it should happen the same irrespective of OS.

What steps will reproduce the problem?

1. This code reproduces the problem 100% of the time here:

   from selenium import webdriver
   from selenium.webdriver.firefox.webdriver import FirefoxBinary
   from selenium.webdriver.common.action_chains import ActionChains
   
   # We have to use a Firefox binary for which native events are supported.
   binary = FirefoxBinary("/home/ldd/src/firefox-34.0.5/firefox")
   
   def perform(native):
       caps = {
           "nativeEvents": native
       }
   
       driver = webdriver.Firefox(capabilities=caps, firefox_binary=binary)
   
       print "NATIVE:", driver.desired_capabilities["nativeEvents"]
   
       driver.get("http://output.jsbin.com/xufota/1")
   
       contents = driver.find_element_by_id("contents")
       ActionChains(driver) \
           .click(contents) \
           .perform()
   
       print driver.execute_script("""
       return document.getElementById("log").textContent;
       """)
   
       driver.quit()
   
   
   for native in (False, True):
       perform(native)
   

   Save it as issue.py.

2. Edit the script so that the path given for the Firefox binary for which native events can be produce is set to a path that works on your system, and run:

   python ./issue.py

What is the expected output?

NATIVE: False

    Log:
  click in over
NATIVE: True

    Log:
  click in over

What do you see instead?

NATIVE: False

    Log:
  click in contents
NATIVE: True

    Log:
  click in over

The click with native events turned off should register in over rather than in contents.

Observations

What the script tries to simulate is a user clicking on the text Hello. When a real user performs this action, the browser will dispatch the event to the element with the id over. When Selenium runs with native events, it is able to dispatch the event in the same way a real browser does. However, Selenium when running with synthesized events is incapable of simulating what happens when a real user clicks in the browser window.

Setting an explicit z-index on over makes no difference.

Using contents.click() rather than an action chain makes no difference.

I'm reporting bug as requested here. In my opinion, the goal of trying to get synthetic events to stand in for native events is unsustainable. The scenario I have here is just the tip of the iceberg.

[lddubeau]

For the record, I have more cases here showing that synthetic events do not correctly simulate real user interaction with a browser. There are problems with:

• focus and blur events,
• mouseup and mousedown events,
• mouseenter and mouseleaveevents.

[AutomatedTester]

Unfortunately having a unmaintained C++ code that depends on an unsupported SDK is a lot more unsustainable. Using this technique also means that we can never tackle mobile.

I would love to see the test cases for all the scenarios we don't handle as these might be useful for the webdriver spec

[jleyba]

If anyone is interested in hacking on the synthetic events, the relevant code is under the javascript/atoms tree.

Tests are easy to write and debug. In your client run

$ ./go debug-server

Then point a browser at http://localhost:2310/javascript/atoms/test and select a test to run.

[lddubeau]

I'll add the other cases I mentioned earlier later.

From what I understand from the bug report, Selenium happened to be using an unsupported API, which was not made for the purpose of generating native events. What needs to happen is for Firefox to have a supported API dedicated to the purpose of generating native events.

The Problem with Synthetic Events

I'm writing this from the perspective of someone who writes complex web applications and uses Selenium to test them across platforms. These applications have their own ways of doing things, but they rely on popular libraries like Bootstrap, DataTables, Angular, and their associated plugins. The problems I'm about to talk about are those that anyone using these libraries are likely to encounter.

As of Selenium 2.46, if developers who write test suites for complex applications that use libraries like those I mentioned above have to use synthetic events on Firefox, this amounts to having them to develop two related but different suites: one that works with native events, and one that works with synthetic events. It is simply not possible right now, to switch the entire test suite to use synthetic events because some platforms only support native events. (I've just tried, right now, to turn off native events in Chrome: the setting is silently ignored.)

(Could just one suite be written that takes into account the lacunae of synthetic events and yet work whether the events generated are synthetic or native? In some cases yes, but not in all cases.)

Sending Events to the Right Element

When the suite uses synthetic events, unless pains are taken to explicitly send events to the right element, the suite is not going to simulate how a real user would interact with the browser. In the case where element A overlaps element B (which is the case in my issue report here), but A is transparent, and the developer tries to simulate what would happen if the user clicks on B, then the developer has to write the suite so that the click event is sent to A. And then if later in development a mistake makes it so that A is erroneously covered by B or by a new element, then the suite won't catch the event delivery problem because A will still get the synthetic event. This problem would be readily caught by Selenium if native events are used but not with synthetic events.

Generating Events

The developer also has to explicitly generate events that would otherwise be automatically generated by the browser. (The issue I've mentioned with focus and blur is exactly this: they are not generated.) Here again, if the developer has to compensate by generating the synthetic events explicitly, this may hide future problems introduced by erroneous changes to the application being tested. And this means relying on execute_script and execute_script_async because there is no other way to manually generate events like blur.

Should Synthetic Events Be The Standard?

In other words, should native event support would be removed from all platforms? I don't think so.

Different browsers generate events differently. Examples:

• IE11 generates input events when an input element gains or loses focus. Other browsers generally do not generate input events for such changes.
• IE processes el.focus() asynchronously, whereas FF and Chrome process it synchronously.

(I shall note here that if IE is not following such and such spec, it is really neither here nor there when it comes to delivering a product that will work. Sure, in a bug report to MS, I'd point out the divergence from the spec but I still have to deliver a product that will work with IE11. So until MS fixes a divergence from the spec, I still need to detect the problem and deal with it.)

These are from cases where my Selenium test suite failed because, fortunately, native events allowed the browser to behave in exactly the same way it would if a user was using it. For instance, the generation of an input event on focus change was completely unexpected and would most likely not have been caught if synthetic events had been used.

To have synthetic events provide the same level of assurance that differences among browsers are going to be caught when the test suite is run (rather than when a user actually triggers the bug in production), Selenium would have to generate synthetic events in a way that mimics the quirks of different browsers (and different versions of the same browser). Is this something Selenium should take on? I don't think so.

[lddubeau]

I got my wires crossed in my initial bug report regarding version numbers. Sorry about that.

To get the complete output I reported, the script has to be run with Selenium 2.45.0. It is possible to run it with 2.46.1 but the attempt at running with native events will fail. At any rate, in 2.46.1 it is still possible to run the part of the test that runs with synthetic events: the incorrect behavior is the same as in 2.45.0.

[barancev]

Fixed by commit 0eec81d