Leverage URL metrics to reserve space for embeds to reduce CLS

[westonruter]

Fixes #1310

The changes in this pull request grew larger than the strict scope of reducing CLS for embeds since the work to implement this uncovered shortcomings with the current Optimization Detective API. This PR includes the following key pieces:

1. Introduce client-side extensions in the form of script modules which are loaded when the detection logic runs. The client-side script modules are able to register functionality to run as soon as the module is loaded by implementing an exported initialize function. They also are able to register functionality to run just before the URL metric data is being submitted by implementing an exported finalize function which amends the URL metric data as needed. This builds on Allow URL metric schema to be extended #1492 which enabled the URL metric schema to be extended. This PR includes Embed Optimizer extending the URL metric element schema to include a resizedBoundingClientRect key which is populated with the script module via a ResizeObserver which starts observing in the module's initialize function. New script modules are added simply by appending to the array passed to the od_extension_module_urls filter.
2. The gathered URL metric data is now sent when the page is hidden as opposed to when the page is "done" loading. This is essential in order to capture the resized heights of embeds once the user scrolls them into view and they are loaded. This will also open up new opportunities to track INP over the life of the page to include in the submitted URL metric data.
3. New APIs are added to Optimization Detective which Embed Optimizer depends on (e.g. OD_URL_Metric_Group_Collection::get_all_denormalized_elements()). In order to avoid the need to do method_exists() checks and the like, a new od_init action is introduced which passes the Optimization Detective version as its only argument. Embed Optimizer and Image Prioritizer then check this passed version to see if it meets the requirements. If not, then an admin notice is displayed. If yes, then it goes ahead and loads the Optimization Detective extension code.
4. Related to the above, I found that certain plugins initialize their module code after the after_setup_theme action. In proposing embedding Optimization Detective in Elementor (Leverage URL metrics to reserve space for embeds to reduce CLS #1373), I found that modules load at init with priority 0. This makes the current bootstrap code unusable when it is loaded in such a module, necessitating an ugly workaround. Since Optimization Detective doesn't need to bootstrap before init anyway, the solution seems to simply be to move the bootstrapping logic to init as well. This is implemented in this PR by moving to init at priority 9 since the plugin has code that runs at init with the default priority of 10.

Accounting for responsive embeds

Initially I was thinking the optimization logic should wait to set the min-height until there are URL metrics gathered for the smallest breakpoint group. For example, let's say the page is responsive and on desktop an embed is much taller than it is on mobile. If the desktop metrics are collected before mobile metrics are collected, then the min-height could be being set for desktop could be too high for mobile, resulting in a gap below the embed on mobile. For example, consider this tweet on mobile vs desktop:

Mobile (320w)	Desktop

The tweet on mobile here is 574px high whereas on desktop it is 825px, so if desktop metrics were only collected then the result would be a worst case gap of 251px at the bottom before the next content:

A simple way to solve this would not to set the min-height on the figure.wp-block-embed element at all, but rather to create a new style rule with responsive styles that sets multiple min-height styles for the element according to each breakpoint. This is implemented in this PR. For example, the following CSS is added to the head for a Twitter embed:

<style>
@media (max-width: 480px) { #embed-optimizer-a7659db28ecaa36ddee6ae66857dabd8 { min-height: 500px; } }
@media (min-width: 481px) and (max-width: 600px) { #embed-optimizer-a7659db28ecaa36ddee6ae66857dabd8 { min-height: 500px; } }
@media (min-width: 601px) and (max-width: 782px) { #embed-optimizer-a7659db28ecaa36ddee6ae66857dabd8 { min-height: 500px; } }
@media (min-width: 783px) { #embed-optimizer-a7659db28ecaa36ddee6ae66857dabd8 { min-height: 500px; } }
</style>

For follow-up

• Move the init to an even later action. Leverage URL metrics to reserve space for embeds to reduce CLS #1373 (comment)
• Consider renaming nonce to hmac to disambiguate from _wpnonce. Leverage URL metrics to reserve space for embeds to reduce CLS #1373 (comment)

Demo Videos

Before

bad-cls-without-embed-optimizer.webm

After

Once URL metrics have been collected from visitors by Optimization Detective:

good-cls-with-embed-optimizer.webm

[westonruter]

It's not ideal to be constructing this XPath manually.

[westonruter]

I've opened #1407 to explore this, but it's not necessary for this PR to move forward.

[westonruter]

I noticed Query Monitor flagging a PHP deprecation error when loading a page with Optimization Detective and Embed Optimizer active. A TikTok embed is present on the page. I'm getting:

PHP Deprecated: Implicit conversion from float 739.9976196289062 to int loses precision in /var/www/html/wp-content/plugins/optimization-detective/class-od-url-metrics-group-collection.php on line 486

Call stack:

Update: Addressed in #1411.

[westonruter]

I also saw this for some reason:

PHP Notice: OD_HTML_Tag_Processor::get_updated_html(): Unable to append markup to optimization_detective_end_of_body since the bookmark no longer exists. in /var/www/html/wp-includes/functions.php on line 6085

Update: See fix below in #1373 (comment)

[westonruter]

This should be a helper method on OD_HTML_Tag_Processor.

[westonruter]

See prior art in the AMP plugin: https://github.com/ampproject/amp-toolbox-php/blob/78531851c59fa5f306315372719f0064b5542cf4/src/Dom/Element.php#L49-L69

[westonruter]

Scratched code to explore using onCLS to detect when an embed has loaded (and causes a layout shift. Problem: In the case of the Twitter embed, the source node being reported is REMOVED from the DOM, so we cannot see the ancestor which has a data-od-xpath attribute.

Code scratch

	const { onLCP, onCLS } = await import( webVitalsLibrarySrc );

	onCLS(
		( metric ) => {
			for ( const entry of metric.entries ) {
				if (
					entry.entryType === 'layout-shift' &&
					! entry.hadRecentInput
				) {
					console.info( entry );
					for ( const source of entry.sources ) {
						console.info( source.node );
					}
				}
			}
		},
		{
			// This is necessary in order to collect all layout shifts, even those which don't cause a bad CLS score.
			reportAllChanges: true,
		}
	);

[felixarntz]

I'm not sure I understand this. Why is the timing of this (in Image Prioritizer and Embed Optimizer) relevant for other plugins? Are you thinking of extensions to those plugins (rather than extensions to Optimization Detective)?

Another thought: Why can't this use the od_init action?

[westonruter]

Bootstrapping at init is all about the embeddability of these plugins inside of other plugins, which is why this whole bootstrap logic was first devised (#1081, #1159, #1185).

For example, in elementor/elementor#28844 I've created a new module for Elementor that embeds the Optimization Detective plugin and its extensions. Elementor initializes its modules at init priority 0. Since none of the functionality in Optimization Detective needs to be initialized before init, there's no need for the current use of after_setup_theme to initialize. So this postpones initialization to latest possible point which opens the door to be embedded in more places.

[felixarntz]

This still feels like a rather shaky foundation to me. For example, why does it use priority 9?

I think I'm still not following what this is meant to do: I think I understand now why certain code needs to run before this callback, but why does certain other code need to run after this callback? For instance, why couldn't we use init on 999999, or even wp_loaded or a later hook?

[westonruter]

Priority 9 is one less than the default priority of 10, as noted in the comment. There are callbacks added at this default priority in the plugin. So priority 9 is the last.

If we used init with an even later priority, then we'd need to adjust the other init actions' priorities as well in this plugin, for example:

performance/plugins/optimization-detective/storage/class-od-url-metrics-post-type.php

Line 48 in 94fc5e7

add_action( 'init', array( __CLASS__, 'register_post_type' ) );

performance/plugins/embed-optimizer/hooks.php

Line 27 in 94fc5e7

add_action( 'init', 'embed_optimizer_add_hooks' );

If we went with wp_loaded, then we couldn't use the init action in the plugins at all, which as I understand is too late for when post types should be registered.

[felixarntz]

So basically this needs to be called as late as possible, but before any of Optimization Detective's own hooks would fire?

[westonruter]

Yes, exactly. As late as possible, and hopefully after any other possible embedding plugins' modules initialize (e.g. Elementor initializes at init priority 0).

[felixarntz]

Makes sense. Since we have full control over the code in Optimization Detective, we may want to consider moving the plugin's init callbacks to a very late priority, so that we can also push this back even further, while still being on init. That way we can cater for other plugins even better, since I wouldn't find it unreasonable for other plugins to load modules on init with the default priority either.

Given the callbacks from Optimization Detective do not have to run after everything else, maybe we just use a very high number (rather than PHP_INT_MAX which I would only use if it technically has to be the last thing).

We can open a follow up PR for this though, if you agree?

[westonruter]

Sounds good!

[felixarntz]

See other comment.

[westonruter]

See #1373 (comment)

[felixarntz]

Sounds good. For the future, ideally the different plugin modifications would happen in separate PRs too. It's more convenient all in one PR, but would be better for review, plus you could still use sub-PRs like you've done before, to e.g. build something in Embed Optimizer on top of another PR change to the Optimization Detective infrastructure.

[felixarntz]

Looking at this old comment thread, does it still require an update? Asking since urlMetric is still being passed below.

[adamsilverstein]

can we add a link to the plugin update screen here?

[westonruter]

We could, but the user may not have the capability to access the plugin update screen. Plugins may not be updatable on the environment either due to it being a read only filesystem. So these complications would seem to make adding a link maybe more trouble than its worth.

[adamsilverstein]

ah, good point. I was trying to make this more helpful... maybe don't show the message to users that can't upgrade plugins?

[westonruter]

True, but I think this might be overkill. It would be unlikely that someone would update Image Prioritizer but not also update Optimization Detective.

[adamsilverstein]

Nice!

[adamsilverstein]

💚 💚 💚 great to see this use navigator.sendBeacon (I wondered about that previously)

[adamsilverstein]

Great!

[adamsilverstein]

Looks great, left a few questions