Skip to content
This repository has been archived by the owner on Mar 26, 2021. It is now read-only.

Expand tests to modern WordPress versions and PHP versions #46

Merged
merged 5 commits into from
Apr 16, 2018

Conversation

benlk
Copy link
Contributor

@benlk benlk commented Mar 20, 2018

Changes

In .travis.yml, and thus in the automated tests that run on Travis CI:

  • PHP versions now covered: 5.6 through 7.1 (not 7.2, for reasons explained later)
  • WordPress versions now covered: 3.8 through 4.9, at the level of the latest security patch

Why

This PR expands the versions tested because we want to make sure that the plugin is compatible with recent PHP and WordPress versions. The simple way to do that is to enumerate a list of WordPress versions and a list of PHP versions, and then run the tests at each intersection. Not only is this approach inefficient, it causes problems when old versions of WordPress include a test suite that assumes an old version of PHPUnit, which is not available when Travis CI only offers one version of PHPUnit per PHP version.

Based on research in npr/pmp-php-sdk#36,
Travis CI runs tests using the following PHPUnit versions:

  • PHP 5.6: PHPUnit 5.7.23
  • PHP 7: PHPUnit 6.4.3
  • PHP 7.1: PHPUnit 6.4.3
  • PHP 7.2: PHPUnit 7.0.2

Reasons that given versions of WordPress have tests incompatible with travis-ci.org:

  1. There are PHP warnings about deprecated PHP-4-style constructors.
  2. Requires PHPUnit 5 because PHPUnit 6 support was not included until WordPress 4.7.
  3. Unless there's been an update since this trac thread, WordPress tests still assume PHPUnit 6, and Travis runs PHP 7.2 tests using PHPUnit 7
phpunit 5.7 6.4 6.4 7.0
php 5.6 7.0 7.1 7.2
WP wp release year then-current php
3.8 1 1 1 2013
3.9 1 1 1 2014 5.5
4.0 1 1 1 2014 5.5
4.1 1 1 1 2014 5.5
4.2 2 2 2 2015 5.6
4.3 2 2 2 2015 5.6
4.4 2 2 2 2015 7.0
4.5 2 2 2 2016 7.0
4.6 2 2 2 2016 7.0
4.7 2 2 2 2016 7.1
4.8 3 2017 7.1
4.9 3 2017 7.2

Based on research in https://github.com/npr/pmp-php-sdk/issues/36,
Travis CI runs tests using the following PHPUnit versions:

- PHP 5.6: PHPUnit 5.7.23
- PHP 7: PHPUnit 6.4.3
- PHP 7.1: PHPUnit 6.4.3
- PHP 7.2: PHPUnit 7.0.2

This commit reduces the number of tests that are run by more than
two thirds. Before, we were running 4 different PHP versions and
11 diffferent WordPress versions. Many of those combinations were
incompatible, for different reasons:

1. There are PHP warnings about deprecated PHP-4-style
   constructors.
2. Requires PHPUnit 5 because PHPUnit 6 support was not
   included until WordPress 4.7.
3. Unless there's been an update since
   [this trac thread](https://core.trac.wordpress.org/ticket/39822),
   WordPress tests still assume PHPUnit 6, and Travis runs PHP 7.2
   tests with PHPUnit 7

| phpunit | 5.7 | 6.4 | 6.4 | 7.0 |                 |                  |
| php     | 5.6 | 7.0 | 7.1 | 7.2 |                 |                  |
| WP      |     |     |     |     | wp release year | then-current php |
|---------|-----|-----|-----|-----|-----------------|------------------|
| 3.8     |     | 1   | 1   | 1   | 2013            |                  |
| 3.9     |     | 1   | 1   | 1   | 2014            | 5.5              |
| 4.0     |     | 1   | 1   | 1   | 2014            | 5.5              |
| 4.1     |     | 1   | 1   | 1   | 2014            | 5.5              |
| 4.2     |     | 2   | 2   | 2   | 2015            | 5.6              |
| 4.3     |     | 2   | 2   | 2   | 2015            | 5.6              |
| 4.4     |     | 2   | 2   | 2   | 2015            | 7.0              |
| 4.5     |     | 2   | 2   | 2   | 2016            | 7.0              |
| 4.6     |     | 2   | 2   | 2   | 2016            | 7.0              |
| 4.7     |     |     |     | 3   | 2016            | 7.1              |
| 4.8     |     |     |     | 3   | 2017            | 7.1              |
| 4.9     |     |     |     | 3   | 2017            | 7.2              |

Accordingly, this commit removes PHP 7.2 from tests, and replaces the
separate lists of PHP and WordPress versions with one list of selected
PHP and WordPress versions.
Updated version of the table in 09d72a8:

Based on research in npr/pmp-php-sdk#36,
Travis CI runs tests using the following PHPUnit versions:

- PHP 5.6: PHPUnit 5.7.23
- PHP 7: PHPUnit 6.4.3
- PHP 7.1: PHPUnit 6.4.3
- PHP 7.2: PHPUnit 7.0.2

Reasons that given versions of WordPress have tests incompatible
with travis-ci.org:

1. There are PHP warnings about deprecated PHP-4-style
   constructors.
2. Requires PHPUnit 5 because PHPUnit 6 support was not
   included until WordPress 4.7.
3. Unless there's been an update since
   [this trac thread](https://core.trac.wordpress.org/ticket/39822),
   WordPress tests still assume PHPUnit 6, and Travis runs PHP 7.2
   tests with PHPUnit 7

| phpunit | 5.7 | 6.4 | 6.4 | 7.0 |                 |                  |
| php     | 5.6 | 7.0 | 7.1 | 7.2 |                 |                  |
| WP      |     |     |     |     | wp release year | then-current php |
|---------|-----|-----|-----|-----|-----------------|------------------|
| 3.8     |     | 1   | 1   | 1   | 2013            |                  |
| 3.9     |     | 1   | 1   | 1   | 2014            | 5.5              |
| 4.0     |     | 1   | 1   | 1   | 2014            | 5.5              |
| 4.1     |     | 1   | 1   | 1   | 2014            | 5.5              |
| 4.2     |     | 2   | 2   | 2   | 2015            | 5.6              |
| 4.3     |     | 2   | 2   | 2   | 2015            | 5.6              |
| 4.4     |     | 2   | 2   | 2   | 2015            | 7.0              |
| 4.5     |     | 2   | 2   | 2   | 2016            | 7.0              |
| 4.6     |     | 2   | 2   | 2   | 2016            | 7.0              |
| 4.7     |     | 2   | 2   | 2   | 2016            | 7.1              |
| 4.8     |     |     |     | 3   | 2017            | 7.1              |
| 4.9     |     |     |     | 3   | 2017            | 7.2              |
@benlk benlk changed the title Improve test coverage in all WordPress versions Expand tests to modern WordPress versions and PHP versions Mar 20, 2018
Copy link

@jgarlow jgarlow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Thanks for the thorough explanation.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants