Navigation Block: Fix erroneous escaping of ampersands (etc) #59561

ockham · 2024-03-04T15:58:44Z

What?

Fix erroneous escaping of ampersands to u0026amp;.

Why?

To fix #59516. Since these "entities" are missing a leading backslash, it them to show up verbatim on the frontend!

How?

By using the rest_pre_insert_{$this->post_type} filter rather than the rest_insert_wp_navigation action. This avoids calling wp_update_post twice, which was the original reason of the issue, as it removed the backslash from the already-encoded entity.

(Note that the fact that encoded entities are sent over the wire in the first place is due to a quirk in the Navigation block: #41063.)

Testing Instructions

(See CI, or run npm run test:php locally.)

This is currently failing, but apparently not because of #59516:

1) Block_Navigation_Block_Hooks_Test::test_block_core_navigation_update_ignore_hooked_blocks_meta Failed asserting that two strings are identical. --- Expected +++ Actual @@ @@ -'' +''

(Note that & is being escaped to \u0026amp;, which seems correct. Most importantly, the leading slash is retained.)

ockham · 2024-03-04T16:03:15Z

cc/ @tjcafferkey @t-hamano

phpunit/blocks/block-navigation-block-hooks-test.php

ockham · 2024-03-05T08:35:56Z

Okay, I've now managed to fix the assertions, and to add a new one that actually fails and captures the problem in #59516.

This should hopefully make it easier to track down the problem.

ockham · 2024-03-05T08:36:34Z

(Oops, fat-fingered the Close button, and then "Delete branch" 😅 )

ockham · 2024-03-05T08:51:14Z

Okay, I've now managed to fix the assertions, and to add a new one that actually fails and captures the problem in #59516.

This should hopefully make it easier to track down the problem.

I also just reverted 467aade (i.e. #59021) locally and ran the tests again. This makes the first two assertions pass (and only the post meta related one fail, as expected), confirming that the unit test covers the bug 👍

ockham · 2024-03-05T09:39:07Z

Here are some tests I ran against https://github.com/WordPress/wordpress-develop:

diff --git a/tests/phpunit/tests/post/getTheContent.php b/tests/phpunit/tests/post/getTheContent.php
index 91f0b621f3..d9030080db 100644
--- a/tests/phpunit/tests/post/getTheContent.php
+++ b/tests/phpunit/tests/post/getTheContent.php
@@ -84,4 +84,24 @@ class Tests_Post_GetTheContent extends WP_UnitTestCase {
 
 		$this->assertSame( 'Foo', get_the_content() );
 	}
+
+	public function test_the_content_block_attribute_escaping_1() {
+		$unescaped = '<!-- wp:navigation-link {"label":"News & About","type":"page","id":2,"url":"http://localhost:8888/?page_id=2","kind":"post-type"} /-->';
+		$escaped   = str_replace( '&', '\u0026amp;', $unescaped );
+
+		$post = self::factory()->post->create_and_get( array( 'post_content' => $unescaped ) );
+
+		$actual = get_the_content( null, true, $post );
+		$this->assertSame( $escaped, $actual );
+	}
+
+	public function test_the_content_block_attribute_escaping_2() {
+		$unescaped = '<!-- wp:navigation-link {"label":"News & About","type":"page","id":2,"url":"http://localhost:8888/?page_id=2","kind":"post-type"} /-->';
+		$escaped   = str_replace( '&', '\u0026amp;', $unescaped );
+
+		$post = self::factory()->post->create_and_get( array( 'post_content' => $escaped ) );
+
+		$actual = get_the_content( null, true, $post );
+		$this->assertSame( $escaped, $actual );
+	}
 }

(Note that test_the_content_block_attribute_escaping_1 creates the post object with the & unescaped, whereas test_the_content_block_attribute_escaping_2 creates it with the & replaced by \u0026amp;.)

This results in:

There was 1 failure:

1) Tests_Post_GetTheContent::test_the_content_block_attribute_escaping_2
Failed asserting that two strings are identical.
--- Expected
+++ Actual
@@ @@
-'<!-- wp:navigation-link {"label":"News \u0026amp; About","type":"page","id":2,"url":"http://localhost:8888/?page_id=2","kind":"post-type"} /-->'
+'<!-- wp:navigation-link {"label":"News u0026amp; About","type":"page","id":2,"url":"http://localhost:8888/?page_id=2","kind":"post-type"} /-->'

/var/www/tests/phpunit/tests/post/getTheContent.php:105

I think this means that we need to run wp_update_post with the & unescaped, as it appears that slashes are otherwise stripped from the content.

This is backed up by a finding that @tjcafferkey just relayed to me: https://github.com/WordPress/wordpress-develop/blob/7002bce8abe6f6f1858bdd2f02cd1168ef50e4d8/src/wp-includes/post.php#L4532

ockham · 2024-03-05T09:48:31Z

Heads-up @getdave @youknowriad -- this is a WIP fix for https://core.trac.wordpress.org/ticket/60671, which is IMO problematic enough that we should get it into RC1. I think we're pretty close to resolving it (within the next 1-2 hours 🤞); it will then require a package sync. Does that work for y'all?

youknowriad · 2024-03-05T09:54:39Z

(within the next 1-2 hours 🤞);

That seems very tight, because we'll need one hour more to do the package release commit approximately and the commit freeze is approaching. Worst case, it lands in RC2.

ockham · 2024-03-05T10:08:21Z

(within the next 1-2 hours 🤞);

That seems very tight, because we'll need one hour more to do the package release commit approximately and the commit freeze is approaching. Worst case, it lands in RC2.

Gotcha, thank you! I'll keep you posted on our progress.

ockham · 2024-03-05T10:10:39Z

I think I have an idea how to solve this. I don't think there's a function in WP to _un_escape stuff like \u0026amp;, so instead, let's try to find a filter to hook block_core_navigation_update_ignore_hooked_blocks_meta into (rather than an action). This should allow us to avoid the extraneous call to wp_update_post.

(We'll still call update_post_meta from the filter, which is probably okay to have as a side-effect.)

ockham · 2024-03-05T11:10:00Z

The fix should be essentially working now. I'll have to update the unit test to reflect the changes (plus the PR title and description).

packages/block-library/src/navigation/index.php

github-actions · 2024-03-05T11:32:33Z

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

Unlinked Accounts

The following contributors have not linked their GitHub and WordPress.org accounts: @kylekelly.

Contributors, please read how to link your accounts to ensure your work is properly credited in WordPress releases.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Unlinked contributors: kylekelly.

Co-authored-by: ockham <bernhard-reiter@git.wordpress.org>
Co-authored-by: t-hamano <wildworks@git.wordpress.org>
Co-authored-by: tjcafferkey <tomjcafferkey@git.wordpress.org>
Co-authored-by: youknowriad <youknowriad@git.wordpress.org>
Co-authored-by: draganescu <andraganescu@git.wordpress.org>
Co-authored-by: annezazu <annezazu@git.wordpress.org>
Co-authored-by: fabiankaegy <fabiankaegy@git.wordpress.org>
Co-authored-by: getdave <get_dave@git.wordpress.org>
Co-authored-by: swissspidy <swissspidy@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

tjcafferkey

This change tests well for me, thanks @ockham! 🙏🏻

One TODO for later I think we can do to make this more clear is documenting the differences between the purposes of the incoming $post (the stdClass) which is being prepared to be written to the DB vs the get_post( $post->ID ) (the WP_Post class) which is being retrieved from the DB. The latter mainly (I say mainly because its also used to retrieve meta) being used to provide context to the Block Hooks API

ockham · 2024-03-05T11:39:06Z

Note to self: Unit tests aside, we should verify that this also fixes https://core.trac.wordpress.org/ticket/60671.

draganescu · 2024-03-05T12:21:18Z

Why does using the rest_insert_wp_navigation action result in calling wp_update_post twice?

ockham · 2024-03-05T12:56:28Z

Why does using the rest_insert_wp_navigation action result in calling wp_update_post twice?

See the diff.

The reason we needed it in the first place was that we needed to persist the post content to the DB, after having inserted hooked blocks. We originally chose the rest_insert_wp_navigation action as it gave direct access to the post object that had been created previously, which allowed us to avoid this logic.

However, since the duplicated wp_update_post call has the nasty side effect of messing up entities, it's preferable to go with the filter (which allows operating on the post content before it is written to the DB, thus limiting the number of wp_update_post calls to only one).

Fix erroneous escaping of ampersands to `u0026amp;`. This is done by using the [`rest_pre_insert_{$this->post_type}`](https://developer.wordpress.org/reference/hooks/rest_pre_insert_this-post_type/) _filter_ rather than the `rest_insert_wp_navigation` _action_. This avoids calling `wp_update_post` twice, which was the original reason of the issue, as it removed the backslash from the already-encoded entity. Unlinked contributors: kylekelly. Co-authored-by: ockham <bernhard-reiter@git.wordpress.org> Co-authored-by: t-hamano <wildworks@git.wordpress.org> Co-authored-by: tjcafferkey <tomjcafferkey@git.wordpress.org> Co-authored-by: youknowriad <youknowriad@git.wordpress.org> Co-authored-by: draganescu <andraganescu@git.wordpress.org> Co-authored-by: annezazu <annezazu@git.wordpress.org> Co-authored-by: fabiankaegy <fabiankaegy@git.wordpress.org> Co-authored-by: getdave <get_dave@git.wordpress.org> Co-authored-by: swissspidy <swissspidy@git.wordpress.org>

getdave · 2024-03-11T11:41:23Z

This fix is critical to ensure correct behaviour of the Navigation block in WP 6.5. Without this blocks would start outputting erroneous characters on the front of the site which would constitute a bug in 6.5 but also break backwards compatbility with previous releases.

I was also identified during the RC 1 phase.

For these reasons I believe this PR is suitable for inclusion in WP 6.5 during the RC phase.

Fix erroneous escaping of ampersands to `u0026amp;`. This is done by using the [`rest_pre_insert_{$this->post_type}`](https://developer.wordpress.org/reference/hooks/rest_pre_insert_this-post_type/) _filter_ rather than the `rest_insert_wp_navigation` _action_. This avoids calling `wp_update_post` twice, which was the original reason of the issue, as it removed the backslash from the already-encoded entity. Unlinked contributors: kylekelly. Co-authored-by: ockham <bernhard-reiter@git.wordpress.org> Co-authored-by: t-hamano <wildworks@git.wordpress.org> Co-authored-by: tjcafferkey <tomjcafferkey@git.wordpress.org> Co-authored-by: youknowriad <youknowriad@git.wordpress.org> Co-authored-by: draganescu <andraganescu@git.wordpress.org> Co-authored-by: annezazu <annezazu@git.wordpress.org> Co-authored-by: fabiankaegy <fabiankaegy@git.wordpress.org> Co-authored-by: getdave <get_dave@git.wordpress.org> Co-authored-by: swissspidy <swissspidy@git.wordpress.org>

getdave · 2024-03-11T14:52:20Z

I just cherry-picked this PR to the pick/wp-65-rc-2 branch to get it included in the next release: b455d76

Fix erroneous escaping of ampersands to `u0026amp;`. This is done by using the [`rest_pre_insert_{$this->post_type}`](https://developer.wordpress.org/reference/hooks/rest_pre_insert_this-post_type/) _filter_ rather than the `rest_insert_wp_navigation` _action_. This avoids calling `wp_update_post` twice, which was the original reason of the issue, as it removed the backslash from the already-encoded entity. Unlinked contributors: kylekelly. Co-authored-by: ockham <bernhard-reiter@git.wordpress.org> Co-authored-by: t-hamano <wildworks@git.wordpress.org> Co-authored-by: tjcafferkey <tomjcafferkey@git.wordpress.org> Co-authored-by: youknowriad <youknowriad@git.wordpress.org> Co-authored-by: draganescu <andraganescu@git.wordpress.org> Co-authored-by: annezazu <annezazu@git.wordpress.org> Co-authored-by: fabiankaegy <fabiankaegy@git.wordpress.org> Co-authored-by: getdave <get_dave@git.wordpress.org> Co-authored-by: swissspidy <swissspidy@git.wordpress.org>

Navigation Block: Add basic test coverage for Block Hooks

e5b6b78

ockham added [Block] Navigation Affects the Navigation Block [Feature] Block hooks labels Mar 4, 2024

ockham self-assigned this Mar 4, 2024

ockham added the [Type] Bug An existing feature does not function as intended label Mar 4, 2024

ockham commented Mar 4, 2024

View reviewed changes

ockham mentioned this pull request Mar 4, 2024

Ampersand's in the Navigation block are converted to u0026 #59516

Closed

ockham commented Mar 4, 2024

View reviewed changes

phpunit/blocks/block-navigation-block-hooks-test.php Outdated Show resolved Hide resolved

ockham added 3 commits March 5, 2024 09:14

Fix post meta assertion

2528541

Get assertion to pass

911d113

Add assertion for get_the_content()

d08b830

ockham closed this Mar 5, 2024

ockham deleted the add/navigation-block-hooks-unit-test branch March 5, 2024 08:35

ockham restored the add/navigation-block-hooks-unit-test branch March 5, 2024 08:36

ockham reopened this Mar 5, 2024

ockham added the Backport to WP 6.7 Beta/RC Pull request that needs to be backported to the WordPress major release that's currently in beta label Mar 5, 2024

ockham added 2 commits March 5, 2024 11:45

Use filter instead of action to avoid calling wp_update_post twice

89ecc65

Pass WP_Post object as context

6b263d4

ockham changed the title ~~Navigation Block: Add basic test coverage for Block Hooks~~ Navigation Block: Fix erroneous escaping of ampersands (etc) Mar 5, 2024

ockham commented Mar 5, 2024

View reviewed changes

packages/block-library/src/navigation/index.php Outdated Show resolved Hide resolved

Set accepted_args to default of 1

97f900e

ockham marked this pull request as ready for review March 5, 2024 11:32

ockham requested a review from ajitbohra as a code owner March 5, 2024 11:32

tjcafferkey approved these changes Mar 5, 2024

View reviewed changes

ockham added 2 commits March 5, 2024 12:48

Fix unit test

68fe053

Skip test on old WP versions

b0a04e9

tjcafferkey mentioned this pull request Mar 5, 2024

Block Hooks API: Add Templates Controller filter to avoid triggering wp_update_post WordPress/wordpress-develop#6225

Closed

ockham merged commit 9f934bb into trunk Mar 5, 2024
56 checks passed

ockham deleted the add/navigation-block-hooks-unit-test branch March 5, 2024 12:59

github-actions bot added this to the Gutenberg 17.9 milestone Mar 5, 2024

ockham mentioned this pull request Mar 11, 2024

Epic: Block Hooks Features for WP 6.5 #54904

Closed

16 tasks

getdave added Backported to WP Core Pull request that has been successfully merged into WP Core and removed Backport to WP 6.7 Beta/RC Pull request that needs to be backported to the WordPress major release that's currently in beta labels Mar 11, 2024

getdave mentioned this pull request Mar 11, 2024

Cherry Pick changes for WordPress 6.5 RC2 #59756

Merged

ockham mentioned this pull request Mar 25, 2024

Navigation: Avoid content loss when only specific entity fields are edited #60071

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Navigation Block: Fix erroneous escaping of ampersands (etc) #59561

Navigation Block: Fix erroneous escaping of ampersands (etc) #59561

ockham commented Mar 4, 2024 •

edited

Loading

github-actions bot commented Mar 4, 2024

ockham Mar 4, 2024

ockham commented Mar 4, 2024

ockham commented Mar 5, 2024

ockham commented Mar 5, 2024

ockham commented Mar 5, 2024

ockham commented Mar 5, 2024

ockham commented Mar 5, 2024

youknowriad commented Mar 5, 2024 •

edited

Loading

ockham commented Mar 5, 2024

ockham commented Mar 5, 2024

ockham commented Mar 5, 2024

github-actions bot commented Mar 5, 2024 •

edited

Loading

tjcafferkey left a comment •

edited

Loading

ockham commented Mar 5, 2024

draganescu commented Mar 5, 2024

ockham commented Mar 5, 2024

getdave commented Mar 11, 2024

getdave commented Mar 11, 2024

+              		$post = get_post( self::$navigation_post );
+              		gutenberg_block_core_navigation_update_ignore_hooked_blocks_meta( $post );
+              		$this->assertSame( self::$original_markup . '<!-- wp:tests/my-block /-->', $post->post_content );

Navigation Block: Fix erroneous escaping of ampersands (etc) #59561

Navigation Block: Fix erroneous escaping of ampersands (etc) #59561

Conversation

ockham commented Mar 4, 2024 • edited Loading

What?

Why?

How?

Testing Instructions

Related

github-actions bot commented Mar 4, 2024

ockham Mar 4, 2024

Choose a reason for hiding this comment

ockham commented Mar 4, 2024

ockham commented Mar 5, 2024

ockham commented Mar 5, 2024

ockham commented Mar 5, 2024

ockham commented Mar 5, 2024

ockham commented Mar 5, 2024

youknowriad commented Mar 5, 2024 • edited Loading

ockham commented Mar 5, 2024

ockham commented Mar 5, 2024

ockham commented Mar 5, 2024

github-actions bot commented Mar 5, 2024 • edited Loading

Unlinked Accounts

tjcafferkey left a comment • edited Loading

Choose a reason for hiding this comment

ockham commented Mar 5, 2024

draganescu commented Mar 5, 2024

ockham commented Mar 5, 2024

getdave commented Mar 11, 2024

getdave commented Mar 11, 2024

ockham commented Mar 4, 2024 •

edited

Loading

youknowriad commented Mar 5, 2024 •

edited

Loading

github-actions bot commented Mar 5, 2024 •

edited

Loading

tjcafferkey left a comment •

edited

Loading