Block API: Server-side awareness of block types

[aduth]

Previously: #2529, #104
Related: #886
Related Slack conversation: https://wordpress.slack.com/archives/C5UNMSU4R/p1505677036000104 (cc @jasonbahl)

When initially developing the API for implementing a block (#104), there were some competing objectives:

• The editor representation of a block must occur in the client to preserve an ideal user experience
• A block should ideally be easy to implement for quick prototyping (Simple API for defining a block #27, e.g. defining block in a browser console)
• General information about a block should be context unaware (title, attributes definition, category, etc need not specifically be defined in a client, server, or an other external static file)

Where we have fallen short is in:

• The third of these objectives; As a need has arisen for server-side attribute validation (Block API: Support and bootstrap server-registered block attribute schemas #2529), we have further fragmented the definition of a block (its attributes may be defined on the server or in the client).
• It is not possible for the server to have reliable awareness of all blocks which would be registered in the editor.

Proposal:

To improve consistency, we should move block attributes (and other general information) to a server block registration. This would likely behave exactly as it does in #2529, including client-side attributes bootstrapping and validation, but would be applied consistently across all blocks. Therefore, every block would have a companion PHP file associated with it, which in addition to current supports defines:

• Title
• Icon
• Category
• Attributes

The trade-off here is that it is not as simple to implement a block in a single location. We could potentially leave client-defined attributes support as-is, but this could also entice developers to avoid the server registration and reintroduce all the inconsistencies/drawbacks therein.

Some previous discussion had considered the idea of a third JSON "manifest" file, similar to a package.json or composer.json, describing the static general information about a block. While this would be a simple format for what is essentially overview data, is problematic because (a) it is yet another file that a developer could need to become familiar with, (b) it is difficult to bring into the client without a build step understanding JSON imports, and (c) localization of strings would need to implement some amount of "magic" (hard-coding to properties expected to be localized).

It's worth pointing out that a plugin author already needs some amount of server-specific logic for each of their blocks, specifically that of enqueueing the JavaScript files responsible for registering the blocks in the client. By requiring block registration in the server, there may be some added benefit in handling scripts and styles.

• A block could define its companion scripts and styles as properties of the block type (e.g. script_handle, style_handle).
• Or: These files could be "discovered" adjacent the block registering file by naming conventions (e.g. find style.css, edit.css, block.js adjacent myblock/index.php). This could encourage developers to define their blocks in a standalone folder/file structure for easier separation overview.

A remaining challenge in moving all block attributes definitions to the server is support for attribute sources. With advent of additional source types (meta, options), we will likely need to expand this concept to cover more than just DOM-based attribute sourcing. Therefore, I would propose creating an alternative structure for representing what currently exists as the assignment of source functions:

Before:

{
	url: {
		source: attr( 'img', 'src' )
	}
}

After:

[
	'url' => [
		'source' => [
			'type'     => 'attribute',
			'selector' => 'img',
			'name'     => 'src',
		],
	],
]

The intent here is not necessarily that the server would become responsible for parsing attributes from the markup of a block (although there is nothing that precludes it from doing so), but rather to be able to represent the structure in a static format, which could then be bootstrapped into the client to recreate the current behavior.

As an example for how this extends into the concept of additional sources:

'author' => [
	'type' => 'string',
	'source' => [
		'type' => 'meta',
		'name' => 'author', // Inferred from attribute name if not specifically defined?
	],
],
'siteTitle' => [
	'type' => 'string',
	'source' => [
		'type' => 'option',
		'name' => 'name',
	],
]

Open questions:

• While there is justification for defining block attributions in the server for the purpose of improving consistency, there is still a lack of general understanding of specific use cases for server-specific awareness of blocks, aside from a "nice to have" reasonings. Lack of these use cases is not intended as an argument for a server definition not to exist, but it would help guide specific implementation decisions.

cc @westonruter

[jasonbahl]

@aduth thanks for creating this ticket and CCing me!

there is still a lack of general understanding of specific use cases for server-specific awareness of blocks, aside from a "nice to have" reasonings. Lack of these use cases is not intended as an argument for a server definition not to exist, but it would help guide specific implementation decisions.

I'll share some use cases (some might still be considered nice to have but in my opinion should not be dismissed).

General WordPress Consistency

WordPress thrives on the hook/filter system, and defining many pieces of the block Schema on just the client limits the power of the system that essentially makes WordPress what it is.

I know there's been work on a JS implementation of similar hook/filter functionality, but that's only useful in the context of WordPress itself, and WordPress is used to power much more than just itself (I'll talk more about decoupled apps below).

Take a step back and imagine if the entire admin was rewritten in JS (think Calypso). And we statically defined Post Type and Taxonomy Registration only in JS and not on the server.

Now think about how much of the WP Ecosystem at large would not exist at all because of the lack of a server-side registration. . .a LOT. There would be no way to generate per-post-type feeds or REST endpoints. Calypso wouldn't be able to work with external WordPress sites, because it wouldn't know what post_types and taxonomies exist in the sites, because that info wouldn't be available via REST, because the REST server wouldn't have any knowledge of it. . .

Without comprehensive server-side registration, we drastically reduce the power of WordPress as we know it to interact with the blocks.

Same goes with most extendable parts of WordPress. . .image sizes, post_stati, post_types, taxonomies, admin_menus, etc. . .so many things are statically defined on the server, so not having a solid registration for Blocks server-side seems like it goes agains WP's history. I know there is a minimal schema, but think if register_post_type was just register_post_type('post') and didn't accept any additional args for labels, capabilities, etc. . .

Also consider existing conversations about how troublesome Meta Boxes in Gutenberg are going to be, mostly due to the lack of a true Fields API (server side schema for fields).

Ideally, one should be able to register a block on the server and that becomes the source of truth for the block's capabilities. The REST API could make use of it, other plugins could make use of it, other APIs (XML-RPC, WPGraphQL, etc) could make use of it, and of course the Gutenberg JS itself could make use of it.

Alternative UIs
Let's be honest. Gutenberg is not going to solve everyones's problems. No matter how awesome it is or will become, it simply will not meet everyone's needs. There will still be a market for creating content in different ways. Page builders currently completely dismiss the Post Edit screen, and these alternative post creation solutions will continue to exist post-Gutenberg.

With a solid Server Side schema, hopefully the Page Builders could at least start to adhere to a standard way of interacting with "Blocks". . .that way, if Page Builder A thinks using Modals to edit content makes more sense than the fixed-right sidebar of Gutenberg, they could build their own UI to interact with Gutenblocks, but still ensure the blocks are saved in the same way.

I think having a solid server-side schema for blocks will help standardize the Page Builder market. Page builders could essentially become "Gutenberg Themes" where the editing experience looks and feels different, but at the end of the day the data produced is the same. This probably isn't impossible without a comprehensive server side schema, but it would be much easier.

If there was a comprehensive Block schema on the server, page builders could use that to hydrate their view layer and ensure compatibility with Gutenberg (avoid the lock-in, at least to a degree, that everyone talks about with page builders).

Headless CMS / Decoupled Apps

Headless CMS with WordPress is nothing new. . .one of my first projects with WordPress was populating a Flash site with data from WordPress via XML-RPC back in ~2009. . .but now with the REST API in core (and exciting plugins like WPGraphQL), it's becoming more and more common to see folks interacting with WordPress in non-traditional Theme / WP-Admin contexts.

For any of these decoupled applications to interact with Gutenblocks, they have to have knowledge of the blocks in the same way Gutenberg itself does. . .not just the blocks that have already been created by Gutenberg inside the WP-Admin, but also the capabilities of what can be done to create/modify blocks. That way these decoupled applications can provide an experience comparable in some fashion to Gutenberg. If you edit a post on a Native Mobile App, Calypso or some other decoupled app, the data should be able to be saved back in the same format that Gutenberg saves it. But in order for said decoupled app to provide an experience that allows for data to be saved in the same format as Gutenberg, said decoupled app needs to know what the capabilities of each block are.

If everything is defined in client-side code that lives in the WP-Admin, we're limiting interaction with the blocks to just the WP-Admin.

The way I see this, is that blocks will have a statically defined registry which will act as the source of truth for what blocks are available, what their capabilities are, the attributes they can have, the types (string, int, bool, array, etc) of data their attributes are expecting to store, etc.

With this knowledge, a decoupled application could request this Block registration from the server, hydrate a client and render out a UI based on the Block registration.

The Gutenberg editor could also use this same registration to Hydrate what blocks are available, what UI elements are needed to provide editing of the available attributes per-block, etc.

Having the configuration defined on the server should also help reduce code duplication. If the server is the source of truth, then the client and server can both use that truth for their needs. The client wouldn't need to define block attributes and fields in addition to the server, just know how to read the source of truth from the server and do something with it, which should actually reduce JS file sizes because there's less info in there defining things, and instead just rendering data that gets served from the client.

Alternate Storage Options
Right now the block data is stored as a large string in the post_content (which is genius for backward compatibility), but I see alternative storage being explored by both core and plugin developers.

For example, where I work we will likely at least experiment with offloading Blocks and their associated meta to ElasticSearch, so that we can read/write individual blocks without having to alter the entire post_content to fix a typo in one block, for example. We will also use ElasticSearch to to aggregations, etc to get insight into how blocks are being used across our sites, etc (how we use ES is beyond the point, though). The point is that the server will need to know about the blocks so validation can be properly made when interacting with external systems.

ElasticSearch has a type mapping for the data that gets sent to it, and if there's a server side schema for blocks, plugins like ElasticPress, or even Jetpack's ElasticSearch implementation would be able to provide mapping for blocks to ES to ensure block data gets indexed properly and efficiently. Without a server side schema, the mapping would be created only as data is sent to ES and ES would give it's best guess on things, but we'd all lose out on the potential ES could give us if there were a server side Block registry that could be used to define mapping.

ElasticSearch aside, I imagine the core team or other developers will explore options for storing Block data in WordPress in some fashion. . .whether it's new wp_blocks and wp_blocks_meta tables or something else, I'm sure folks will experiment with storing blocks outside of a massive post_content string and as we've learned with the evolution of post_meta, the lack of a schema makes things "interesting" to say the least.

WPGraphQL

I'm the creator and maintainer of WPGraphQL (https://github.com/wp-graphql/wp-graphql) which brings a GraphQL API to WordPress. . .so I'll pitch my case, which I think is applicable to all the above in some fashion.

WPGraphQL allows for data to be declaratively fetched and for just the requested data to be sent to the client from the server.

An example of a WPGraphQL query right now would be:

{
  post(id:"someId") {
     id
     title
  }
}

And that produces a JSON response like so:

{
  data: {
    post: {
       id: "someId",
       title: "Hello World"
    }
  }
}

As you can see, unlike REST where a request sends back a pretty large payload with the entire post object (unless you've written up some custom feature endpoints 🤢 ), the minimal amount of data needed is transferred from server to client, which is super beneficial, especially on slow mobile networks where data transfer can hold things up.

Anyway, with Gutenberg I envision WPGraphQL being a super powerful tool, especially for decoupled applications. I can picture folks using Gutenblocks with various block meta to indicate whether a block should be viewable on desktop or mobile, (or in our case WordPress even powers our newspaper print content). With WPGraphQL, a native mobile client could ask for just blocks that are meant for use in a native context.

For example, Let's say I've extended all of my Gutenblocks to have a "Context" multi-select which would allow me to apply various contexts for where my blocks should display (Web / Native Mobile / Print, for example).

I could have a post created by Gutenberg that has this HTML:

<!-- wp:core/heading {"align":"center", contexts:[web]} -->
<h3 style="text-align:center" id="some-anchor">Heading to show only on web</h3>
<!-- /wp:core/heading -->

<!-- wp:core/heading {"align":"center", contexts:[mobile-native, web]} -->
<h3 style="text-align:center" id="some-anchor">Heading to show ONLY on mobile</h3>
<!-- /wp:core/heading -->

I could query from a React native (or similar) app, and ask for just the blocks that were tagged with "mobile-native" context. . .something to this tune:

{
   post(id:"someId") {
      id
      title
      contentBlocks( where: { context: MOBILE } ) {
           ...on HeadingBlock {
               rawContent
               __typename
           }
      }
   }
}

I would receive a payload like:

{
  data: {
    post: {
       id: "someId",
       title: "Hello World",
       contentBlocks: [
         {
             rawContent: "<h3 style="text-align:center" id="some-anchor">Heading to show ONLY on mobile</h3>",
              __typename: "HeadingBlock"
          }
       ]
    }
  }
}

My post contains multiple blocks, but my Native client was able to request just the blocks that were intended to be rendered in the Native context. This reduces network bandwidth as the data send to the client is limited to what should be sent to the client, reduces bugs in code as it's a schema with defined type system, reduces transformation on the client as the payload contains what's needed so the client doesn't need to parse/transform the Gutenblocks for all contexts to determine which ones it should use, etc. . .

With the current implementation of gutenberg_parse_blocks my hypothetical scenario is already partly possible with WPGraphQL, but the big missing piece (in this example), would be WPGraphQL knowing what Enum options there are for the Context select. . . WPGraphQL would have no knowledge of what options exist for the Context select field if it's defined in the client, not the server. It wouldn't even know that "context" was a possible attribute of a block/all blocks if the attributes weren't defined on the server.

=====

I'll stop here, as I probably just sound like I'm spouting a bunch of nonsense. . .but happy to elaborate, discuss, clarify anything I've pointed out. . .and who knows, maybe some things I've pointed out already have been addressed since the last time I had to dig through things??

[westonruter]

@aduth I highly support what you've proposed here. By having a language-agnostic schema defined for what a block's attributes look like, this can only open opportunities for improvements for portability of blocks across systems. If source types can all be defined in a schema then it would open the possibility for a server-side PHP processor to be able to parse and apply transformations to a block, or to parse a block in order to extract data for an indexing service, for example. It could also then be used in mobile apps for native code to do the same. The blocks API written in JS then in Gutenberg would serve as a reference implementation of a standardized/common format.

[sc0ttkclark]

+1 if we could register blocks via PHP, this would greatly improve the ability for plugins like Pods to register blocks etc. Otherwise we're stuck building our own interface for it that then outputs JS objects that we have another JS file loaded to register those JS objects as blocks in the editor.

[sc0ttkclark]

After a short discussion, what we want for Pods integration would be PHP registration of blocks, with some way for us to provide JS callbacks for edit/save, so that we can do what we need dynamically. In addition to this, having an ability to hook into a Gutenberg save-specific filter (what data to save) or action (saving) would be very handy for us to do some of the more complicated feature integrations that Pods would need.

[aduth]

If we move toward registering all blocks on the server for a context-unaware definition of a block, what's left to implement in the client editor is to support the user interactions therein: edit is the most obvious. Others like toolbar controls and inspector are included here, though currently as an implementation detail but would otherwise generally belong in the client-specific behaviors as well. Less obvious are details like: icon, category, transforms, and maybe even save. I'm inclined to treat these as client-specific (despite the text of my original comment). save is a can of worms I'm hesitant to explore: The roles it serves in the browser are important to enabling compatibility / reusability toward an edit and save duality, validating blocks, and generating markup for the text editing modes. Defining this on the server would be harmful to all these points, however exposes the possibility of handling block content updates from non-Gutenberg clients.

[aduth]

Related Slack conversation: https://wordpress.slack.com/archives/C02QB2JS7/p1507560212000456

[joehoyle]

Seems like this should happen sooner than later, as I'd imagine many are already taking advantage of the JavaScript registration API to add support in plugins etc - it would be a shame to cause duplicative work for any 3rd parties.

Is anyone owning this / have a write up on specifically what we need to do? If the code is written, perhaps it's mainly a documentation effort needed next?

[aduth]

The most recent work here is in #2854, merged in the past week. Effectively we should have all of the pieces to start moving block registration to the server, while still supporting fully client-side block capabilities (there's still some desire to enable a client editor to be fully functional without a server part).

[westonruter]

I see this being similar to what Customizer does with registering controls in core. They usually get registered in PHP but this is just a wrapper that passes the parameters that you passed to the client for passing into the corresponding JS API for registration. This gives opportunities for PHP plugins to know about and manipulate the controls, and the same would be true for blocks.

[aduth]

I see this being similar to what Customizer does with registering controls in core. They usually get registered in PHP but this is just a wrapper that passes the parameters that you passed to the client for passing into the corresponding JS API for registration

Effectively this does already exist, currently limited to attributes but easily expandable to any / all block properties we want to support from server registration:

gutenberg/lib/client-assets.php

Lines 747 to 757 in 7f518a4

	// Preload server-registered block schemas.
	$block_registry = WP_Block_Type_Registry::get_instance();
	$schemas = array();
	foreach ( $block_registry->get_all_registered() as $block_name => $block_type ) {
	if ( isset( $block_type->attributes ) ) {
	$schemas[ $block_name ] = $block_type->attributes;
	}
	}
	wp_localize_script( 'wp-blocks', '_wpBlocksAttributes', $schemas );

[jasonbahl]

@aduth you pointed out that this preloads server-registered blocks:

gutenberg/lib/client-assets.php

Lines 747 to 757 in 7f518a4

	// Preload server-registered block schemas.
	$block_registry = WP_Block_Type_Registry::get_instance();
	$schemas = array();
	foreach ( $block_registry->get_all_registered() as $block_name => $block_type ) {
	if ( isset( $block_type->attributes ) ) {
	$schemas[ $block_name ] = $block_type->attributes;
	}
	}
	wp_localize_script( 'wp-blocks', '_wpBlocksAttributes', $schemas );

However, it's a completely optional thing. Gutenberg still loads up all 40 (or however many) blocks whether there are any blocks registered server side or not.

With plugin developers being asked to migrate functionality to Gutenblocks, the server-side API needs to be more of a first-class citizen.

Let's take Matias demo from WCUS as an example. He showcased editing a "Book" page that had some content and an image.

Let's say the workflow for an organization is that certain user roles can edit the content, and certain user roles can edit the image.

If all block logic is handled on the client, that means any user role can effectively edit any content, as the restriction to what they can edit is just a client-side prop that could easily be changed via the browser console giving any user power to do whatever they want. . .

The server needs to be the contract between the client and the persistence layer. We can't trust the client 💯 .

I get that the current TinyMCE is basically free-reign, and any content can be placed in there without server-side validation, but that's 1 big text input. Gutenberg is introducing 100's of new text inputs, but not introducing server-side validation/auth checks for any of them.

I feel like for a long time, when learning best practices folks would say: "Look how core does it", but when folks "look how core does it" and see Gutenberg doing things exclusively on the client with no contract on the server determining what's safe and expected, folks will follow suit, and will start writing code that trusts the client explicitly, encouraging bad practices and all sorts of security vulnerabilities down the road.

If Gutenberg is trying to be "the best editor" out there, and "leapfrog medium and squarespace", etc. . .then let's make it "the best editor" by also making sure it adheres to best practices (never trust the client, handle validation on the server, etc)

[westonruter]

It would be interesting to see block-level capabilities. In the classic editor, if an admin adds a script tag or iframe and saves the post, but then an editor comes along and updates that same post, the script/iframe will get stripped by Kses.

In Gutenberg, the Custom HTML block could be used by an admin to add a script tag or iframe. If an editor user comes along and edits the post, they should be able to do so without destroying that block's contents. This could be done by sanitizing/validating block-by-block rather than the post_content as a whole. There are a couple server side attributes here: capability, validate_callback, and sanitize_callback. If the user doesn't have the required capability, then the block should be read only and locked. If the validate_callback returns a WP_Error for a given block's attributes, then the post save could be rejected from the REST API. And then the sanitize_callback would be essentially by default just do Kses like core does already.

[aduth]

Some further exploring of expanding the server bootstrapping at #3962

[spacedmonkey]

I have made a first pass for this API in a core trac ticket here. Can someone review and get back to me?

[gziolo]

I have made a first pass for this API in a core trac ticket here. Can someone review and get back to me?

Awesome, I will have a look closer to the end of the week 👍

[spacedmonkey]

@aduth @gziolo Decided to move the patch for the block type REST API to a gutenberg PR for easier testing. To that end, I have created #21065 .

Can anyone interested please take a look and see if this REST API has the required fields and data. Remember we can add fields later.

[gziolo]

Thank you @spacedmonkey and @TimothyBJacobs for making it happen!

[mtias]

Great to see this one in :)