Playground compatibility tester

[bgrgicak]

We recently built a prototype of a Playground compatibility tester for WordPress.org plugins. Let's talk about what features the Playground compatibility tester should include.

Some suggestions that came up in conversations with @adamziel and @akirk:

• Automatically test the top 1000 (or another number) of WordPress.org plugins.
• Test all plugins locally and add daily automated tests for recently updated plugins.
• Collect stats from each run to understand how changes to Playground are impacting compatibility (number of PHP and SQL errors over time).
• Create a report page for each plugin in markdown with the list of the latest errors and links to reproduce errors in Playground.
• Use git versioning to store test runs instead of storing all runs in timestamp folders.
• Add support for multiple blueprints. This would allow plugin owners and us to create different test scenarios.
• Add more tests. We currently only test if Playground can boot with a blueprint, but we could also test if it responds to an HTTP request or even run basic E2E tests like loading WP-admin.
• Support themes.
• Create a guide for plugin owners on how to integrate Playground tester into their CI pipeline. This will allow plugin developers to automatically test changes.
• Use the test results to create PHPCS rules to help developers avoid unsupported features. We could try generating this with AI.
• Integrate with QIT.

cc @WordPress/playground-maintainers @akirk

[bgrgicak]

The V1 of the plugin tester is completed .
After running boot tests all logs are parsed and used to generate daily reports.

• Playground Error Report
• SQL Errors Report
• PHP Errors Report

[adamziel]

@bgrgicak Let's make this work eventually visible in the main Playground repo for anyone looking for this kind of compatibility information. Could be docs, could be a README entry, could be something else. CC @juanmaguitar

[bgrgicak]

Ok, but I need a few days to polish all automation bugs, so once we add it to our documentation we can "forget" about it.

[bgrgicak]

@juanmaguitar would you have time to work on these docs? The tester should be stable next week.

[bgrgicak]

@adamziel suggested

"If this could have one more column for “active plugin installs” and be sorted by that in a descending way, I’d say that’s all we need for the SQL work today."

"For the platform-level PHP work, support for themes and that one HTTP request would go a long way."

[asirota]

Seems like this project could go a long way to run continuous plugin-check plugin to test for continuous compliance to be in the WordPress repository.

[bgrgicak]

I would love that, but it first needs to scale to support all plugins, and we need reports for each individual plugin.

I'm working on these features, but it's a low priority for me, so it will take some time.

[asirota]

Yeah I hear you. This project has so many implications for WordPress. It's so awesome to watch it evolve. Amazing work everyone!