-
Notifications
You must be signed in to change notification settings - Fork 4.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Workflows: Run PHP unit tests also against current WP - 1 #46983
Conversation
581ed06
to
3afff6a
Compare
.github/workflows/phpunit-tests.yml
Outdated
include: | ||
# Test with the previous WP version. | ||
- php: '7.4' | ||
wordpress: '6.0' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In a follow-up, we could consider computing the previous WP series automatically (i.e. figure out that 6.1 is the current series, and that thus, 6.0 is the previous one).
.github/workflows/phpunit-tests.yml
Outdated
|
||
env: | ||
WP_ENV_PHP_VERSION: ${{ matrix.php }} | ||
WP_ENV_CORE: WordPress/WordPress${{ matrix.wordpress != '' && format( '#{0}-branch', matrix.wordpress ) || '' }} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The advantage of using the WordPress/WordPress
git repo's x.y-branch
is that we don't have to compute the latest .patch version in that series (would be 6.0.3 here).
Flaky tests detected in a0e62f3. 🔍 Workflow run URL: https://github.com/WordPress/gutenberg/actions/runs/4607133381
|
The newly added WP 6.0 test is currently failing with
Does that ring a bell for anyone? Some kind of wordpress-phpunit version incompatibility? Some missing env var or argument for |
2ce991c
to
2a371d8
Compare
e2d7611
to
945ca4e
Compare
Looks like that the file that So we might want a mechanism to also fetch a different version of |
Ah, IIUC, per #41780, |
Right on... I do remember this issue, but not sure where I saw it before. Probably in that PR! I guess it'd be worth adding loggind and debug information related to wp-env. Idea: wp-env could be caching something and not updating. It attempts to run from cache to be fast. However, the cache buster should take into account at a minimum We could verify the logic to ensure that it correctly updates if the global variable changes. We could also run |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for submitting the PR!
Please find my questions below.
.github/workflows/unit-test.yml
Outdated
include: | ||
# Test with the previous WP version. | ||
- php: '7.4' | ||
wordpress: '6.0' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does that mean this string need to be changed when a new WordPress comes out and the previous version gets outdated? If so, is there a way to automate it (so no manual changes are needed)? I'm asking just out of curiosity.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, I would like to automate this (and actually even had written a comment on it that was probably somewhat lost in rebases). I didn't want to make it a blocker for this PR though as it might get a bit involved.
(We'd likely get the latest
version from this WordPress.org REST API endpoint, truncate the point version to get the "series", and then infer the previous series from it. E.g. 6.0.3
-> 6.0
-> 5.9
. Requires a bit of version arithmetics that I thought would be a nice fit for a separate PR. We have some similar logic e.g. here.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It’s just a thought, but what if wpenv
supported something like
"core": "WordPress/WordPress#previous-stable",
or just
"core": "WordPress/WordPress#previous",
?
This way we wouldn't have to write any custom scripts and it will work everywhere (not just in GitHub CI workflows).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it's a good idea, but we'd probably need different syntax. That basically just converts directly to git -- e.g. github.com/WordPress/Wordpress, with #
corresponding to a git ref (like a tag)
945ca4e
to
063f5d0
Compare
I will do my best to test this PR today/tomorrow. Thanks. |
@noahtallen I just tried this in 063f5d0, but unfortunately, tests are still failing with the same error. |
063f5d0
to
61b4027
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please find my comment below. Thanks.
.github/workflows/unit-test.yml
Outdated
|
||
env: | ||
WP_ENV_PHP_VERSION: ${{ matrix.php }} | ||
WP_ENV_CORE: WordPress/WordPress${{ matrix.wordpress != '' && format( '#{0}-branch', matrix.wordpress ) || '' }} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've tested this PR, and I don't think -branch
needs to be appended to the name of the WP version.
When I log in to the docker container, I get 6.0.4-alpha-55372
as the WP version in src/wp-includes/version.php
, but it should be 6.0
instead.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure I agree 😅 In the example you're giving, I'd want us to use 6.0.3
. That's currently the latest version from the 6.0
series that has all the bugfixes that were deemed critical enough to warrant a point release (well, three of them); and I believe that the auto-updater would update most people's WordPress installation to 6.0.3
, if they had previously installed 6.0
(or 6.0.1
or 6.0.2
).
Using -branch
is a cheap way of getting that latest version (that doesn't require us to do any REST API lookups). Now since we're fetching our WordPress from git, we're not getting 6.0.3
exactly, but the latest state of the 6.0
branch, which might have a few commits on top; and which has a weird 6.0.4-alpha-something
version tag, as you've found.
I still find that preferable over using 6.0
. Why? Imagine there is a bug in 6.0
that causes a problem (manifested by a unit test error) if the Gutenberg plugin is active. We can fix that bug, and it could make its way into a 6.0.x
point release, but it'd never get into 6.0
! This means the tests just would keep failing.
The optimal solution would likely be to actually compute the latest version from the 6.0
series (via something like this). Personally, I'd be fine keeping the -branch
approach for a first iteration, and do a separate PR to do the version number computation later. Would that be okay?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see your point, and I agree it's better to use the latest minor release (if available).
But, I think using 6.0-branch
is less optimal than 6.0.3
because regular WordPress users don't usually update their websites to the latest "branch" version.
They update to the latest minor or major release using the WordPress dashboard.
From my point of view, it makes more sense to test Gutenberg with WP versions that most users can use and update to.
Personally, I'd be fine keeping the -branch approach for a first iteration, and do a separate PR to do the version number computation later. Would that be okay?
Yes, I think that is fine. Thanks.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just re-iterating that this PR isn't currently working -- we still need to fix this issue 😬
I think I know why it's happening. if ( ! wpVersion || wpVersion.match( /-(?:alpha|beta|rc)/i ) ) {
ref = 'trunk';
fetchRaw.push( 'fetch', 'origin', ref, '--depth', '1' );
} else {
ref = `tags/${ wpVersion }`;
fetchRaw.push( 'fetch', 'origin', 'tag', wpVersion, '--depth', '1' );
}
} Debug output:
So it's fetching trunk despite the value of the |
I've investigated why these test fail, @ockham. The same applies to the error in P.S. I've just asked myself what we are really doing here. |
Thank you very much for investigating @anton-vlasenko !
Hmm 🤔 My mental model was a bit different here: We're running old WordPress code plus current Gutenberg code, which includes the "compatibility layer" code that includes versions of classes (and functions) such as cc/ @oandregal since the code (and tests) in question seems to fall into your area of expertise 😊 |
👋 I see the (I'm looking at the other failing test. I'm in the middle of a work trip, so I don't know how far I'll be able to dig into this this week). |
@ockham I've taken a look at the test. #45634 was the PR that introduced it. It backported WordPress/wordpress-develop#3517 that was the one that introduced the change. As far as I could understand, the change there was to remove the existing caching in the resolver and make dependand on another cache that exists in wp_query. I don't know when this wp_query cache was introduced. If it was introduced in 6.1 but it's not present in 6.0, it'd explain the difference. I'm pinging the original authors of the PR in case they can provide any advice @peterwilsoncc @hellofromtonya @andrewserong In any case, it sounds like |
Thanks for the ping! This is a really interesting one, and I think this PR has actually revealed something we missed during the backporting process of some of the layout changes for 6.1. The TL;DR is that for Gutenberg to be properly compatible with WordPress 6.0, we missed adding in a couple of required CSS properties ( More background:
So, from the sounds of it, for this test #49118 should fix it up. And the good thing is that it's revealed the value of adding in a job to tests against the previous WP version! I don't think we would have caught this issue otherwise. |
Update: I've merged in #49118, so after a rebase, this PR should hopefully pass the |
Thank you so much @oandregal and @andrewserong! It'd be amazing if this PR already helped unearth a compatibility issue with WP 6.0 😄 Gonna rebase now; crossing my fingers that the test will pass afterwards 🤞 |
8bd8f52
to
1b95e3c
Compare
Looks like the The
|
Highlighting this suggestion from @oandregal :
|
Since WP 6.2 is scheduled to be released tomorrow, let's wait a bit longer and then rebase this PR to see what happens once WP-1 becomes 6.1. Maybe that will resolve our remaining open issue -- hopefully it will not introduce a new one 🤞 |
1b95e3c
to
1a22fbe
Compare
🎉 🎉 🎉 Anyone wanna give approval to merge this? 😊 |
This reverts commit 61b4027.
Co-authored-by: Peter Wilson <519727+peterwilsoncc@users.noreply.github.com>
e63b159
to
3dd4a46
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice! Seems to be working well. One concern is just how many of these get added to each PR. Not sure how GitHub actions allocates boxes to CI builds, but could see us reaching some kind of resource limit. At some point, it might make sense to try to optimize the speed of each build
Thank you, Noah!
Yeah, indeed. I'm slightly concerned about the added jobs as well, but I guess let's see how it pans out and if we can optimize things a bit 😊 |
test-php: | ||
name: PHP ${{ matrix.php }}${{ matrix.multisite && ' multisite' || '' }} on ubuntu-latest | ||
name: PHP ${{ matrix.php }}${{ matrix.multisite && ' multisite' || '' }}${{ matrix.wordpress != '' && format( ' (WP {0}) ', matrix.wordpress ) || '' }} on ubuntu-latest |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A problem with having the WordPress version in the name is that each time the version changes, someone has to update the required jobs in the repo settings. This slack thread discusses a situation where it happened today:
https://wordpress.slack.com/archives/C02QB2JS7/p1719295939361199
It'd be good to think about possible solutions to this. Only including the major version number (e.g. 6.4.x) might be one option.
What?
Run PHP unit tests also against the previous version of WordPress.
Fixes #46979.
Why?
tl;dr: Because Gutenberg is supposed to be compatible with the latest WP version and with the one before it, but sometimes breaks when installed on the latter
See #46979 for more background.
How?
Extends the PHP unit test matrix so that when run with PHP 7.4, tests are not only run for the current WordPress version, but also for the one before.
Testing Instructions
See CI.