djangocon-2019-web-ui-testing

This repository contains the companion project for the Hands-On Web UI Testing tutorial taught by Pandy Knight at DjangoCon 2019. If you are taking the tutorial, then please clone this repository and follow the instructions below. The slide deck for the tutorial is also in this repository.

WARNING!

This project is archived. It was the example code for a one-time workshop, and it will not be actively maintained. If you want to learn how to do Web UI testing with Python and Selenium WebDriver, then please take Selenium WebDriver with Python from Test Automation University. It is a free course that covers the same material as this tutorial.

Python Setup

You can complete this tutorial using any OS: Windows, macOS, Linux, etc.

This tutorial requires Python 3.7 or higher. You can download the latest Python version from Python.org.

This tutorial also requires pipenv. To install pipenv, run pip install pipenv from the command line.

You should also have a Python editor/IDE of your choice. Good choices include PyCharm and Visual Studio Code.

You will also need Git to copy this project code. If you are new to Git, try learning the basics.

WebDriver Setup

For Web UI testing, you will need to install the latest versions of Google Chrome and Mozilla Firefox. You can use other browsers with Selenium WebDriver, but the tutorial will use Chrome and Firefox.

You will also need to install the latest versions of the WebDriver executables for these browsers: ChromeDriver for Chrome and geckodriver for Firefox. Each test case will launch the WebDriver executable for its target browser. The WebDriver executable will act as a proxy between the test automation and the browser instance. Please use the latest versions of both the browsers and the WebDriver executables. Older versions might be incompatible with each other.

ChromeDriver and geckodriver must be installed on the system path.

To install them on Windows:

1. Create a folder named C:\Selenium.
2. Move the executables into this folder.
3. Add this folder to the Path environment variable. (See How to Add to Windows PATH Environment Variable.)

To install them on Linux, macOS, and other UNIX variants, simply move them to the /usr/local/bin/ directory:

$ mv /path/to/ChromeDriver /usr/local/bin
$ mv /path/to/geckodriver /usr/local/bin

This directory should already be included in the system path. For troubleshooting, see:

• Setting the path on macOS
• Setting the path on Linux

To verify correct setup on any operating system, simply try to run them from the terminal:

$ ChromeDriver
$ geckodriver

You may or may not see any output. Just verify that you can run them without errors. Use Ctrl-C to kill them.

Project Setup

1. Clone this repository.
2. Run cd djangocon-2019-web-ui-testing to enter the project.
3. Run pipenv install to install the dependencies.
4. Run pipenv run python -m pytest to verify that the framework can run tests.
5. Create a branch for your code changes. (See Repository Branching below.)

Repository Branching

The master branch contains the code for the tutorial's starting point. The project is basically empty in the master branch.

If you want to code along with the tutorial, then create a branch for your work off the master branch. To create your own branch named tutorial/develop, run:

> git checkout master
> git branch tutorial/develop
> git checkout tutorial/develop

The example/* branches contain the completed code for tutorial parts. If you get stuck, you can always check the example code.

• example/1-first-test
• example/2-webdriver-setup
• example/3-page-objects
• example/4-locators
• example/5-webdriver-calls
• example/6-browser-config
• example/7-race-conditions
• example/8-parallel-testing
• example/develop (main development branch for the examples)

Tutorial Instructions

Part 1: Writing Our First Test

Time Estimate: 5 Minutes

Example Branch: example/1-first-test

We should always write test cases before writing any test code. Test cases are procedures that exercise behavior to verify goodness and identify badness. Test code simply automates test cases. Writing a test case first helps us form our thoughts well.

Consider the following test case:

Scenario: Basic DuckDuckGo Search
    Given the DuckDuckGo home page is displayed
    When the user searches for “panda”
    Then the search result title contains “panda”
    And the search result query is “panda”
    And the search result links pertain to “panda”

Let's implement this test using pytest. Create a new file named test_search.py under the tests directory, and add the following code:

"""
These tests cover DuckDuckGo searches.
"""

def test_basic_duckduckgo_search():

    # Given the DuckDuckGo home page is displayed
    # TODO

    # When the user searches for "panda"
    # TODO

    # Then the search result title contains "panda"
    # TODO
    
    # And the search result query is "panda"
    # TODO
    
    # And the search result links pertain to "panda"
    # TODO

    raise Exception("Incomplete Test")

Adding comments to stub each step may seem trivial, but it's a good first step when writing new test cases. We can simply add code at each TODO line as we automate. Once we finish writing the test's code, we will remove the exception at the end. Also, note that pytest expects all test functions to begin with test_.

To avoid confusion when we run tests, let's remove the old placeholder test. Delete tests/test_fw.py.

Rerun the tests using pipenv run python -m pytest. The test_basic_duckduckgo_search should be the only test that runs, and it should fail due to the "Incomplete Test" exception.

Finally, commit your code change. Part 1 is complete!

Part 2: Setting Up Selenium WebDriver

Time Estimate: 5 Minutes

Example Branch: example/2-webdriver-setup

Selenium WebDriver is a tool for automating Web UI interactions with live browsers. It works with several popular programming languages and browser types.

The Selenium WebDriver package for Python is named selenium. Run pipenv install selenium to install it for our project.

Every test should use its own WebDriver instance. This keeps things simple and safe. The best way to set up the WebDriver instance is to use a pytest fixture. Fixtures are basically setup and cleanup functions. As a best practice, they should be placed in a conftest.py module so they can be used by any test.

Create a new file named tests/conftest.py and add the following code:

"""
This module contains shared fixtures.
"""

import pytest
import selenium.webdriver


@pytest.fixture
def browser():

  # Initialize the ChromeDriver instance
  b = selenium.webdriver.Chrome()

  # Make its calls wait up to 10 seconds for elements to appear
  b.implicitly_wait(10)

  # Return the WebDriver instance for the setup
  yield b

  # Quit the WebDriver instance for the cleanup
  b.quit()

The browser fixture uses Chrome. Other browser types could be used instead. Real-world projects often read browser choice from a config file here.

The implicit wait will make sure WebDriver calls wait for elements to appear before sending calls to them. 10 seconds should be reasonable for this test project's needs. For larger projects, however, setting explicit waits is a better practice because different calls need different wait times. Read more about implicit versus explicit waits here.

The yield statement makes the browser fixture a generator. The first iteration will do the "setup" steps, while the second iteration will do the "cleanup" steps. Each test must make sure to quit the WebDriver instance as part of cleanup, or else zombie processes might lock system resources!

Now, update test_basic_duckduckgo_search in tests/test_search.py to call the new fixture:

def test_basic_duckduckgo_search(browser):
  # ...

Whenever a pytest test function declares a fixture by name as an argument, pytest will automatically call that fixture before the test runs. Whatever the fixture returns will be passed into the test function. Therefore, we can access the WebDriver instance using the browser variable!

Rerun the test using pipenv run python -m pytest to test the fixture. Even though the test should still fail, Chrome should briefly pop up for a few seconds while the test is running. Make sure Chrome quits once the test is done. Then, commit your latest code changes. Part 2 is now complete!

Part 3: Defining Page Objects

Time Estimate: 10 Minutes

Example Branch: example/3-page-objects

A page object is an object representing a Web page or component. They have locators for finding elements, as well as interaction methods that interact with the page under test. Page objects make low-level Selenium WebDriver calls so that tests can make short, readable calls instead of complex ones.

Since we have our test steps, we know what pages and elements our test needs. There are two pages under test, each with a few interactions:

1. The DuckDuckGo search page
   • Load the page
   • Search a phrase
2. The DuckDuckGo results page
   • Get the result link titles
   • Get the search query
   • Get the title

Let's write stubs for our page object classes. Each interaction should have its own method. Later, we can implement the interaction methods with Selenium WebDriver calls. Create a new Python package named pages. To do this create a directory under the root directory named pages. Then, put a blank file in it named __init__.py. The pages directory should not be under the tests directory. Why? When using pytest, the tests folder should not be a package.

Create a new module named pages/search.py and add the following code for the DuckDuckGo search page:

"""
This module contains DuckDuckGoSearchPage,
the page object for the DuckDuckGo search page.
"""


class DuckDuckGoSearchPage:

  def __init__(self, browser):
    self.browser = browser

  def load(self):
    # TODO
    pass

  def search(self, phrase):
    # TODO
    pass

Create another new module named pages/result.py and add the following code for the DuckDuckGo result page:

"""
This module contains DuckDuckGoResultPage,
the page object for the DuckDuckGo search result page.
"""


class DuckDuckGoResultPage:
  
  def __init__(self, browser):
    self.browser = browser

  def result_link_titles(self):
    # TODO
    return []
  
  def search_input_value(self):
    # TODO
    return ""

  def title(self):
    # TODO
    return ""

Every page object needs a reference to the WebDriver instance. That's why the __init__ methods take in and store a reference to browser.

Finally, update test_basic_duckduckgo_search in tests/test_search.py with the following code:

"""
These tests cover DuckDuckGo searches.
"""

from pages.result import DuckDuckGoResultPage
from pages.search import DuckDuckGoSearchPage


def test_basic_duckduckgo_search(browser):
  search_page = DuckDuckGoSearchPage(browser)
  result_page = DuckDuckGoResultPage(browser)
  PHRASE = "panda"
  
  # Given the DuckDuckGo home page is displayed
  search_page.load()

  # When the user searches for "panda"
  search_page.search(PHRASE)

  # Then the search result title contains "panda"
  assert PHRASE in result_page.title()
  
  # And the search result query is "panda"
  assert PHRASE == result_page.search_input_value()
  
  # And the search result links pertain to "panda"
  for title in result_page.result_link_titles():
    assert PHRASE.lower() in title.lower()

  # TODO: Remove this exception once the test is complete
  raise Exception("Incomplete Test")

Notice how we are able to write all the test steps using page object calls and assertions. We also kept the step comments so the code is well-documented. Even though we haven't made any Selenium WebDriver calls, our test case function is nearly complete! Our code is readable and understandable. It delivers clear testing value.

Rerun the test using pipenv run python -m pytest. The test should fail again, but this time, it should fail on one of the assertions. Then, commit your latest code changes. Part 3 is now complete!

Part 4: Finding Locators for Elements

Time Estimate: 15 Minutes

Example Branch: example/4-locators

An element is a "thing" on a Web page. Browsers render elements such as buttons, dropdowns, and input fields using the page's HTML code. Users interact directly with the page's elements. Tests use page objects to interact with elements like a user.

Interactions typically require three steps:

1. Wait for the target element to exist
2. Get an object representing the target element
3. Send commands to the element object

In our solution, waiting is handled automatically thanks to the browser fixture's implicitly_wait call. Getting the element object, however, requires a locator.

Locators are query strings that use HTML attributes to find elements on a Web page. There are many types of locators:

• ID
• Name
• Class name
• CSS Selector
• XPath
• Link text
• Partial link text
• Tag name

For example, if the page has the following element:

<button id="django_ok">OK</button>

Then, a page object could use an ID locator for "django_ok" to get this element.

Locators are not element objects themselves but instead point to elements. The WebDriver instance uses locators to fetch and construct element objects. Why are locators and elements separate concerns? Elements on a page are always changing: they may take time to load, or they may change with user interaction. Locators, however, are always the same: they simply specify how to get elements. For example, a locator could be used to prove that an element does not exist.

For our test, we need locators for three elements:

1. The search input on the DuckDuckGo search page
2. The search input on the DuckDuckGo results page
3. The result links on the DuckDuckGo results page

(Note: The page title is not a Web element. It can be fetched as a browser property.)

Writing good locators is a bit of an art. Inspecting the HTML source of a live page makes it easy. To do this, open the DuckDuckGo search page in Chrome. Then, right-click the page and select "Inspect". Chrome DevTools will open. The "Elements" tab shows the HTML source. As you move the cursor over HTML elements in the source, Chrome will highlight the elements on the page. Now, click the icon with the box and cursor in the upper-left corner of the DevTools pane. Move the cursor over elements on the page, and you will see them highlighted in the source. Neat!

Try to find the search input element. Its HTML should look like this:

<input
  id="search_form_input_homepage"
  class="js-search-input search__input--adv"
  type="text"
  autocomplete="off"
  name="q"
  tabindex="1"
  value=""
  autocapitalize="off"
  autocorrect="off">

Notice that there is an id attribute set to "search_form_input_homepage". Let's use this as our locator and update pages/search.py.

First, import By from the selenium package so we can write locators:

from selenium.webdriver.common.by import By

Then, add the following attribute to the DuckDuckGoSearchPage class:

SEARCH_INPUT = (By.ID, 'search_form_input_homepage')

By contains property keys for each type of locator. We can write locators as tuples of the locator type and the query string. (We will use this locator for interaction calls in the next part of the tutorial.)

The full code for pages/search.py should now look like this:

"""
This module contains DuckDuckGoSearchPage,
the page object for the DuckDuckGo search page.
"""

from selenium.webdriver.common.by import By


class DuckDuckGoSearchPage:

  SEARCH_INPUT = (By.ID, 'search_form_input_homepage')

  def __init__(self, browser):
    self.browser = browser

  def load(self):
    # TODO
    pass

  def search(self, phrase):
    # TODO
    pass

Let's write locators for the DuckDuckGoResultPage next. Perform a search, inspect the page, and try to come up with locators on your own.

Below is the code for pages/result.py with locators:

"""
This module contains DuckDuckGoResultPage,
the page object for the DuckDuckGo search result page.
"""

from selenium.webdriver.common.by import By


class DuckDuckGoResultPage:
  
  RESULT_LINKS = (By.CSS_SELECTOR, 'a.result__a')
  SEARCH_INPUT = (By.ID, 'search_form_input')
  
  def __init__(self, browser):
    self.browser = browser

  def result_link_titles(self):
    # TODO
    return []
  
  def search_input_value(self):
    # TODO
    return ""

  def title(self):
    # TODO
    return ""

Thankfully, the search input element on the result page is similar to the one on the search page.

The locator for the result links is a bit trickier, though. It must find all result elements that contain the search phrase in their display texts. This locator will return a list of elements, not just one. The result links are all "a" hyperlink elements with a class named "result__a". We can use a CSS selector for its query.

We should always try to use the simplest locator that uniquely finds the target elements. IDs, names, and class names are the easiest, but sometimes, we must use CSS selectors and XPaths. To learn more about writing good locators, take the Web Element Locator Strategies course from Test Automation University.

Although the test will still fail, rerun it using pipenv run python -m pytest to make sure our changes did no harm. Then, commit your latest code changes. Part 4 is now complete!

Part 5: Making WebDriver Calls

Time Estimate: 15 Minutes

Example Branch: example/5-webdriver-calls

Now we can implement all the page object methods using WebDriver calls. The WebDriver API for Python documents all WebDriver calls. If you aren't sure how to do something, look it up. WebDriver can do anything a user can do on a Web page!

Let's start with DuckDuckGoSearchPage. The load method is a one-line WebDriver call, but it's good practice to make the URL a class variable:

URL = 'https://www.duckduckgo.com'

def load(self):
  self.browser.get(self.URL)

The search method is a bit more complex because it interacts with an element. We need to use a locator to find the search input element, and then we need to send keys to type the search phrase into the element.

First, update the selenium package imports:

from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys

The search method needs two parts: finding the element and sending the keystrokes. Thankfully, we already have the locator for the element.

def search(self, phrase):
  search_input = self.browser.find_element(*self.SEARCH_INPUT)
  search_input.send_keys(phrase + Keys.RETURN)

The find_element method will return the first element found by the locator. Notice how the locator uses the * operator to expand the SEARCH_INPUT locator tuple into arguments. The selenium package offers specific locator type methods (like find_element_by_name), but using the generic find_element method with argument expansion is better practice. If the locator type must be changed due to Web page updates, then the find_element call would not need to be changed.

The send_keys method sends the search phrase passed into the search method. This means that the page object can search any phrase! The addition of Keys.RETURN will send the ENTER/RETURN key as well, which will submit the input value to perform the search and load the results page.

The full code for pages/search.py should look like this:

"""
This module contains DuckDuckGoSearchPage,
the page object for the DuckDuckGo search page.
"""

from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys


class DuckDuckGoSearchPage:

  # URL

  URL = 'https://www.duckduckgo.com'

  # Locators

  SEARCH_INPUT = (By.ID, 'search_form_input_homepage')

  # Initializer

  def __init__(self, browser):
    self.browser = browser

  # Interaction Methods

  def load(self):
    self.browser.get(self.URL)

  def search(self, phrase):
    search_input = self.browser.find_element(*self.SEARCH_INPUT)
    search_input.send_keys(phrase + Keys.RETURN)

Now, let's do DuckDuckGoResultPage. The title method is the easiest one because it just returns a property value:

def title(self):
  return self.browser.title

The search_input_value method is similar to the search method from DuckDuckGoSearchPage, but instead of sending a command, it asks for state from the page. The "value" attribute contains the text a user types into an "input" element.

def search_input_value(self):
  search_input = self.browser.find_element(*self.SEARCH_INPUT)
  value = search_input.get_attribute('value')
  return value

The result_link_titles method is a bit more complex. The test must verify that the result page displays links relating to the search phrase. This method should find all result links on the page. Then, it should get the titles for those result links. Remember, the test asserts that the search phrase is in each title. This assertion may seem too stringent because it could fail the test for possibly valid links, but it should be good enough for simple search terms. (Again, remember, the test is merely a basic search test.)

The result_link_titles method should look like this:

def result_link_titles(self):
  links = self.browser.find_elements(*self.RESULT_LINKS)
  titles = [link.text for link in links]
  return titles

Notice that it uses find_elements (plural) to get a list of matching elements.

The full code for pages/result.py should look like this:

"""
This module contains DuckDuckGoResultPage,
the page object for the DuckDuckGo search result page.
"""

from selenium.webdriver.common.by import By


class DuckDuckGoResultPage:
  
  # Locators

  RESULT_LINKS = (By.CSS_SELECTOR, 'a.result__a')
  SEARCH_INPUT = (By.ID, 'search_form_input')

  # Initializer

  def __init__(self, browser):
    self.browser = browser

  # Interaction Methods

  def result_link_titles(self):
    links = self.browser.find_elements(*self.RESULT_LINKS)
    titles = [link.text for link in links]
    return titles
  
  def search_input_value(self):
    search_input = self.browser.find_element(*self.SEARCH_INPUT)
    value = search_input.get_attribute('value')
    return value

  def title(self):
    return self.browser.title

Finally, remove the "incomplete" exception from tests/test_search.py. That module's code should look like this:

"""
These tests cover DuckDuckGo searches.
"""

from pages.result import DuckDuckGoResultPage
from pages.search import DuckDuckGoSearchPage


def test_basic_duckduckgo_search(browser):
  search_page = DuckDuckGoSearchPage(browser)
  result_page = DuckDuckGoResultPage(browser)
  PHRASE = "panda"

  # Given the DuckDuckGo home page is displayed
  search_page.load()

  # When the user searches for "panda"
  search_page.search(PHRASE)

  # Then the search result title contains "panda"
  assert PHRASE in result_page.title()
  
  # And the search result query is "panda"
  assert PHRASE == result_page.search_input_value()
  
  # And the search result links pertain to "panda"
  for title in result_page.result_link_titles():
    assert PHRASE.lower() in title.lower()

Rerun the test using pipenv run python -m pytest. Now, finally, it should run to completion and pass! The test will take a few second to run because it must wait for page loads. Chrome should pop up and automatically go through all test steps. Try not to interfere with the browser as the test runs. Make sure pytest doesn't report any failures when it completes.

Part 6: Configuring Multiple Browsers

Time Estimate: 15 Minutes

Example Branch: example/6-browser-config

Our test currently runs on Chrome, but it should be able to run on other browsers, too. Any Web UI test should be configurable to run on any applicable browser. Let's run it on Headless Chrome and Firefox!

Browser choice is an aspect of testing. In theory, every test should run on every supported browser. Thus, browser choice should be treated as an input for test automation. It should not be hard-coded into automation code. It should also not be written as pytest parameters. One test session should use one browser. If another browser needs to be tested, then launch another test session. This design keeps test code and test executions simpler.

Create a new file named config.json in the project's root directory. JSON files are very easy to use in Python. The json module is part of the standard library, and JSON files can be parsed into dictionaries with one line. Add the following lines:

{
  "browser": "Chrome",
  "implicit_wait": 10
}

Notice how these inputs correspond to values in the browser fixture. Then, add a new fixture to tests/conftest.py:

import json

@pytest.fixture
def config(scope='session'):

  # Read the file
  with open('config.json') as config_file:
    config = json.load(config_file)
  
  # Assert values are acceptable
  assert config['browser'] in ['Firefox', 'Chrome', 'Headless Chrome']
  assert isinstance(config['implicit_wait'], int)
  assert config['implicit_wait'] > 0

  # Return config so it can be used
  return config

This fixture reads the config.json file. It also validates the inputs so that tests won't run if the inputs are bad. The fixture's scope is set to "session" so that the fixture is called only one time for all tests. There is no need to read it repeatedly for every test.

Update the browser fixture to use these inputs:

@pytest.fixture
def browser(config):

  # Initialize the WebDriver instance
  if config['browser'] == 'Firefox':
    b = selenium.webdriver.Firefox()
  elif config['browser'] == 'Chrome':
    b = selenium.webdriver.Chrome()
  elif config['browser'] == 'Headless Chrome':
    opts = selenium.webdriver.ChromeOptions()
    opts.add_argument('headless')
    b = selenium.webdriver.Chrome(options=opts)
  else:
    raise Exception(f'Browser "{config["browser"]}" is not supported')

  # Make its calls wait for elements to appear
  b.implicitly_wait(config['implicit_wait'])

  # Return the WebDriver instance for the setup
  yield b

  # Quit the WebDriver instance for the cleanup
  b.quit()

Fixtures can call fixtures. Here, browser calls config and then uses its parts to set the browser and implicit wait time. Notice that Headless Chrome just uses the Chrome WebDriver with extra arguments.

Nothing else needs to be updated in order to change the browser. Run the test using pipenv run python -m pytest with Chrome to verify no harm was done. You should see the test run successfully.

Then, change the config's browser to "Headless Chrome" and rerun the test. You won't see the browser window appear, but the test should still pass. Why? "Headless" mode won't render pages visibly. It's great for automated testing because it's slightly more efficient than "regular" Chrome.

Finally, try "Firefox". Does it work? Warning: it may or may not! Oh no! Don't panic if it doesn't work. We'll fix it in the next part.

Part 7: Handling Race Conditions

Time Estimate: 15 Minutes

Example Branch: example/7-race-conditions

When running the search test using Firefox, you might hit the following failure:

      # Then the search result title contains "panda"
>     assert PHRASE in result_page.title()
E     AssertionError: assert 'panda' in 'DuckDuckGo — Privacy, simplified.'

Or, the test might pass. Why would the test fail on Firefox if it passed for Chrome? And why is there a chance that it might fail or might pass? Let's revisit the test case steps:

Scenario: Basic DuckDuckGo Search
    Given the DuckDuckGo home page is displayed
    When the user searches for “panda”
    Then the search result title contains “panda”
    And the search result query is “panda”
    And the search result links pertain to “panda”

Step 2 performs the search. Then, step 3 checks the title of the page. Unfortunately, step 3 has a race condition. Remember, the browser and the automation are two separate processes. When the automation triggers the search, the browser will load the new page and title. At the same time, the automation will continue to execute the test. If the automation executes the assertion before the new page title loads, then the assertion will fail. Chrome was fast enough to avoid the race condition, but Firefox was slow enough to trigger it.

Race conditions are the bane of Web UI testing. They can be difficult to predict when writing tests. They can also be difficult to identify in test results because they typically happen intermittently. Web UI tests gain a bad reputation for being "flaky" whenever race conditions are not handled appropriately.

Automation must always wait for page components to be ready before interacting with them. Implicit waits work well for Web elements, but they don't work for browser attributes like page title. They are best for small projects. Explicit waits are more customizable, but they require more code. They are typically the better option for large projects that need different times and conditions. As a best practice, automation should use only one type of waiting. Mixing implicit and explicit waits can have unexpected consequences.

Thankfully, there's a shortcut we can use to fix test_basic_duckduckgo_search. The other two assertions use implicit waits for other elements on the page. By the time those elements are loaded, the title would also be loaded. Therefore, we can move step 3 to the end of the scenario to be the last thing we check.

The updated code for tests/test_search.py should be:

"""
These tests cover DuckDuckGo searches.
"""

from pages.result import DuckDuckGoResultPage
from pages.search import DuckDuckGoSearchPage


def test_basic_duckduckgo_search(browser):
  search_page = DuckDuckGoSearchPage(browser)
  result_page = DuckDuckGoResultPage(browser)
  PHRASE = "panda"
  
  # Given the DuckDuckGo home page is displayed
  search_page.load()

  # When the user searches for "panda"
  search_page.search(PHRASE)

  # Then the search result query is "panda"
  assert PHRASE == result_page.search_input_value()
  
  # And the search result links pertain to "panda"
  for title in result_page.result_link_titles():
    assert PHRASE.lower() in title.lower()

  # And the search result title contains "panda"
  # (Putting this assertion last guarantees that the page title will be ready)
  assert PHRASE in result_page.title()

Rerun the test using pipenv run python -m pytest with Firefox to verify the fix. Then, rerun it again with Chrome and Headless Chrome to make sure those browsers still work.

Always watch out for race conditions, always wait for things to be ready before interacting with them, and always run tests multiple times across multiple configurations to identify problems.

Part 8: Running Tests in Parallel

Time Estimate: 15 Minutes

Example Branch: example/8-parallel-testing

Unfortunately, Web UI tests are very slow compared to unit tests and service API tests. The best way to speed them up is to run them in parallel.

First, let's parametrize test_basic_duckduckgo_search so that we have more than one test to run. Any pytest test or fixture may be parametrized. Update the code in tests/test_search.py to be:

"""
These tests cover DuckDuckGo searches.
"""

import pytest

from pages.result import DuckDuckGoResultPage
from pages.search import DuckDuckGoSearchPage


@pytest.mark.parametrize('phrase', ['panda', 'python', 'polar bear'])
def test_basic_duckduckgo_search(browser, phrase):
  search_page = DuckDuckGoSearchPage(browser)
  result_page = DuckDuckGoResultPage(browser)
  
  # Given the DuckDuckGo home page is displayed
  search_page.load()

  # When the user searches for the phrase
  search_page.search(phrase)

  # Then the search result query is the phrase
  assert phrase == result_page.search_input_value()
  
  # And the search result links pertain to the phrase
  for title in result_page.result_link_titles():
    assert phrase.lower() in title.lower()

  # And the search result title contains the phrase
  # (Putting this assertion last guarantees that the page title will be ready)
  assert phrase in result_page.title()

The test will now run three times with different search phrases. Rerun the tests to make sure they all work. You will notice that they run one at a time.

Next, install pytest-xdist, the pytest plugin for parallel testing:

$ pipenv install pytest-xdist

Finally, run the tests using the following command:

$ pipenv run python -m pytest -n 3

The "-n 3" arguments tells pytest to run 3 tests in parallel. We have 3 example tests, and most machines can handle 3 Web UI tests simultaneously. When the tests run, notice how 3 browser instances open at once - one per test.

Run the tests a few times using Chrome and Firefox. Look to see how long the tests typically take per browser. Also, look to see if any intermittent failures happen. Then, try using Headless Chrome. Most likely, Headless Chrome will be significantly faster and more reliable that regular Chrome and Firefox.

As a warning, parallel testing can be dangerous. Make sure that tests avoid collisions. Collisions happen when tests simultaneously access shared state. For example, one test could try to access a database record while another test deletes it. Thankfully, our DuckDuckGo search tests do not have any collisions because they make independent searches in separate browser instances.

Whenever running tests in parallel, carefully tune the number of threads to minimize the total test execution time. More threads does not necessarily mean faster testing. Too many parallel tests will choke system resources.

Anecdotally, for Web UI tests:

• 1 test per processor minimizes total execution time without slowing down individual tests
• 2 tests per processor minimizes total execution time further but slows down individual tests
• more than 2 tests per processor does not meaningfully shrink total execution time further
• memory size does not have a significant impact on total execution time

One machine can scale up only so far. For massive parallel testing, try using Selenium Grid. Alternatively, many companies provide cloud-based solutions for parallel WebDriver testing. Check the Resources section below for a list.

To learn more about parallel testing in general, read To Infinity and Beyond: A Guide to Parallel Testing.

Congrats! You have completed the guided part of this tutorial!

Independent Exercises

The guided tutorial covered one very basic search test, but DuckDuckGo has many more features. Try to write some new tests for DuckDuckGo independently. Here are some suggestions:

• search for different phrases
• search by clicking the button instead of typing RETURN
• click a search result
• expand "More Results" at the bottom of the result page
• verify auto-complete suggestions pertain to the search text
• search by selecting an auto-complete suggestion
• search a new phrase from the results page
• do an image search
• do a video search
• do a news search
• change settings
• change region

These tests will require new page objects, locators, and interaction methods. See how many tests you can automate on your own! If you get stuck, ask for help.

Additional Resources

This DjangoCon 2019 Hands-On Web UI Testing tutorial is related to other tutorials by Andrew Knight:

• PyOhio 2019: Hands-On Web UI Testing
• TestProject: Web Testing Made Easy with Python, Pytest and Selenium WebDriver
• SmartBear: Hands-On UI Testing with Python

Test Automation University offers free online courses on several testing and automation topics. All TAU courses are great, but the following ones compliment this tutorial especially well:

• Web Element Locator Strategies shows how to write good locators and use Chrome DevTools.
• Behavior-Driven Python with pytest-bdd shows how to use pytest-bdd to write BDD-style tests.
• Setting a Foundation for Successful Test Automation shows how to run a testing project the right way.

Many companies provide cloud-based solutions for parallel, multi-browser, multi-platform Web UI testing that work with WebDriver:

• BrowserStack
• LambdaTest
• Sauce Labs
• SmartBear CrossBrowserTesting
• TestProject

Other helpful links:

• AutomationPanda.com
  • Python
  • Testing
  • Why Python is Great for Test Automation
  • Web Element Locators for Test Automation
  • The Testing Pyramid
  • To Infinity and Beyond: A Guide to Parallel Testing
• Selenium with Python
  • WebDriver API
  • Waits
  • Locating Elements
• pytest.org
• Selenium Grid wiki

Special Thanks from Pandy

Thank you to DjangoCon 2019 for inviting me to deliver this tutorial!

Thank you to all the students who participated in this tutorial at DjangoCon 2019.

Thank you to the following individuals who graciously reviewed this tutorial:

• Michael Lynch (@deliberatecoder)
• Satyank Tiwari (@satyanktiwari)
• Adeel Mansoor (@testadeel)
• Rick Clymer (@clymerrm)
• Katrina Durance (@katdurance)

About the Author

This tutorial was written and delivered by Andrew Knight (aka Pandy), the "Automation Panda". Andy is a Pythonista who specializes in testing and automation.

• Twitter: @AutomationPanda
• Blog: AutomationPanda.com
• LinkedIn: andrew-leland-knight