Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RFC 100: Enhancing headless support in Wagtail core #100

Draft
wants to merge 4 commits into
base: main
Choose a base branch
from
Draft

RFC 100: Enhancing headless support in Wagtail core #100

Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
d75cea4
First commit
thibaudcolas Jul 10, 2024
3223379
Update 100-headless-support.md
thibaudcolas Jul 31, 2024
6dcbcbb
Update 100-headless-support.md
thibaudcolas Nov 7, 2024
3757ef4
Update 100-headless-support.md
thibaudcolas Nov 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
83 changes: 83 additions & 0 deletions text/100-headless-support.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
# RFC 100: Headless support in Wagtail core

- RFC: 100
- Author: Thibaud Colas
- Created: 2024-07-31
- Last Modified: 2024-07-31

## Abstract

Wagtail’s support for headless websites should move closer to feature-parity with Django monoliths, without reliance on third-party packages for compatibility with core functionality.
For features that aren’t supported because of technical constraints, design decisions, or inertia, there should be documented workarounds or alternatives.

## Why

- Because a lot of existing and future sites would benefit from more cohesive headless support.
- Because it’s harder to maintain this kind of support across external packages.

## Features to support

Here are two rules to determine which current and future Wagtail features must be supported for headless websites:

1. If a feature is "on by default" in Wagtail core, we must aim to support it with no external package or other custom Django implementation.
2. If a feature is on by default and unsupported, it must be possible to turn it off and we must document a workaround or alternative.

## Feature-by-feature review

Started from feature definitions of [Are we headless yet?](https://areweheadlessyet.wagtail.org/), expanded as appropriate
allcaps marked this conversation as resolved.
Show resolved Hide resolved

| Feature | Goal in Wagtail core | Current status |
| ----------------------------------------------------------------------------------------- | -------------------- | ------------------ |
| [REST API](https://areweheadlessyet.wagtail.org/rest-api) | Full support | Full support |
| [GraphQL](https://areweheadlessyet.wagtail.org/graphql) | No support | External package |
| [Page Preview](https://areweheadlessyet.wagtail.org/page-preview) | Full support | External package |
| [Images](https://areweheadlessyet.wagtail.org/images) | Full support | Partial support |
| [Page URL Routing](https://areweheadlessyet.wagtail.org/page-url-routing) | Full support | Full support |
| [Rich Text](https://areweheadlessyet.wagtail.org/rich-text) | Full support | Known shortcomings |
| [Multi-site support](https://areweheadlessyet.wagtail.org/multi-site-support) | Full support | Known shortcomings |
| [Form submissions](https://areweheadlessyet.wagtail.org/form-submissions) | No support | No support |
| [Password-protected Pages](https://areweheadlessyet.wagtail.org/password-protected-pages) | Full support | No support |
| [Internationalisation](https://areweheadlessyet.wagtail.org/internationalisation) | Full support | TBC |
| [StreamField](https://areweheadlessyet.wagtail.org/streamfield) | Full support | TBC |
| Userbar | Full support | No support |
| Content checks | Full support | No support |
| Redirects | Full support | No support |
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With a headless site, the backend only needs to store the redirects. Wagtail currently does this. The frontend needs to be able to query the redirects from the headless cms and apply them. I would say that wagtail provides what a headless cms needs to provide for handling redirects at this juncture. I have a next.js front end that query's redirects from wagtail at build time and it works just fine.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not entirely sure it does 🤔

I had to use some custom code for finding a page by URL when that page had a redirect: https://github.com/nationalarchives/ds-wagtail/blob/develop/etna%2Fapi%2Furls.py#L162-L195

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't go through the trouble of querying redirects page by page. I just grab all redirects and renderer them as pages that redirect or update the redirect list on my CDN at build time. You might also consider graphql/grapple instead of interacting with the API directly. It's much easier to get all your data per page in a single query that way vs making oodles of rest calls everytime you generate a page.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dopry ah I think you’re right, I didn’t realise there was already a built-in endpoint to fetch the redirects.

@ahosgood if I understand the code you’re sharing, it returns a page that’s associated to a redirect and matches the provided html_path? That feels useful but more than I’d expect is needed to get to feature parity (which is what this RFC is about). In that scenario, it seems like the site’s front-end would return a 200 with the page data for the redirect path?

Copy link

@ahosgood ahosgood Aug 19, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dopry I'm not sure I've understood fully.

Our headless CMS is a separate service to the frontend and they are built and deployed independently so building Wagtail doesn't interact with the frontend code.

Also, redirects can be added at any time by updating a Wagtail slug so I don't think computing all redirects at build time is a possibility for us.

Copy link

@ahosgood ahosgood Aug 19, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@thibaudcolas Essentially, yes.

We had two choices:

  1. Return the page data even when requesting the "old" path (redirect just in the API)
  2. Return the data needed for the frontend to be able to redirect

In the frontend I have made a simple switch for both of these options because I wasn't sure which was going to be best for us. https://github.com/nationalarchives/ds-frontend/blob/main/app/wagtail/routes.py#L231-L239

As you can see, I also had some (now commented out) code to return redirects to external sites as well which is also possible in Wagtail.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ahosgood We generate the redirects when we build the frontend. We're using next.js on Netlify. In our nextjs config we have a function for the next.js redirects key that queries the redirects from grapple. When it get's published on netlify, their Next.JS integration pushes the redirects to their CDN. We also render the redirects as pages just in case for some unforseen reason the redirect isn't in the CDN it's a static page. We are querying redirects through grapple/graphql. @thibaudcolas I'm am not sure that the native wagtail API provides a list of redirects. They are available in grapple/graphql interface.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The API can expose redirects (https://docs.wagtail.org/en/stable/reference/contrib/redirects.html#api) but it sounds like your use case is to query all the redirects and then build your service using them whereas ours is not built statically.

Our frontend just queries the Wagtail API directly so I can add a redirect to Wagtail at any point and the frontend can use it straight away.

The non-headless Wagtail responds to a URI whilst taking redirects into consideration. The API does not (without having to make multiple calls) so I would argue that it is needed for parity.


Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should/can we add the Wagtail accessibility checker in here? It's a really useful tool and while it works in the CMS, it might be a good idea to have the ability to add it to the frontend as well. Of course that also means there will be a requirement for validating logged in Wagtail users in the frontend and including some JS and CSS.

Maybe I'm overthinking it?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe it's part of the "Content checks" and "Userbar" items. The accessibility checker is one of the content checks, and we currently piggyback on the userbar (the Wagtail button in the frontend) to implement it 🙂

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. I’ll update the RFC to make sure it’s more explicit.

## Gaps to address

Copy link
Member

@allcaps allcaps Aug 4, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've numbered my feedback. I hope it helps to keep discussion structured. There is no specific order.

I'm not sure if my points are in line with the goal of this RFC. I just mention them as I've experienced them as points to address in my headless projects.

1. All page URLs endpoint

In a frontend built step, all pages need to be build. Wagtail could use an endpoint listing all page URLs (no other fields needed, without pagination). One call to built them all.

2. Invalidate/purge

Because the frontend acts as a cache, it needs to be invalidated/purged. In Wagtail, it is necessary to track which object is used on which page so that when content is mutated, the correct frontend components/pages are invalidated/purged.

3. OpenAPI specification and/or JSON schema

I really like to have an OpenAPI specification or JSON schema available. This makes frontend development easier. Could Wagtail provide a schema for all its endpoints? Via introspection of Models and streamfields?

FYI: I my headless projects I roll my own DRF endpoints and DRF Spectacular for this.

4. Point Wagtail admin live button to the frontend

Override Page.get_url_parts to point the Wagtail admin live button to the frontend URL.

5. Override Page.serve to serve the API response

I'd like to override the Page serve method to serve the API response. This gives a one-to-one mapping with the frontend application and makes development much easier. I think many headless projects could benefit from this.

class APIMixin:
    def serve(self, request, *args, **kwargs):
        request.is_preview = getattr(request, "is_preview", False)

        serializer = ...

        # A DRF view gives us nice html or json output.
        # The default render class should be listed first.
        @api_view(("GET",))
        @renderer_classes([JSONRenderer, BrowsableAPIRenderer])
        def view(req):
            return Response(serializer(self, context={"request": request}).data)

        return view(request)

This is also a (partial) solution for the userbar. Example: it is easy to switch between https://nl.wagtail.space/ and https://cms.wagtail.space/

6. The way to expose Settings (and Snippets)

Site settings defining a menu, header, footer, contact details, or what not, should be exposed via REST API endpoints. Wagtail could define a best practice.

7. Translation of hardcoded strings

There are strings marked for translation in Django/Wagtail, and Django has the JavaScriptCatalog view. Many frontends come with their own translation solutions.

This might duplicate the way to lookup a specific translation, or lead to duplicated definitions.
If the decision is to use Django JavaScriptCatalog view, frontend translations might end-up in the backend.

8. Forms

Regular Django forms and Wagtail Form Builder forms.

Both need to re-implement rendering in the frontend.
Form validation has logic in backend (to clean/validate the submitted data) and frontend (for direct feedback).

Forms that validate input on-the-fly against the backend and use the error messages supplied by the backend have a single point of truth. To me, this works best.

9. RoutablePageMixins

The RoutablePageMixin mixin provides a convenient way for a page to respond on multiple sub-URLs with different views. The page is the starting point for fixed URL patterns.

I'm not sure how Wagtail API suggests to handle these. I guess point 5 is a possible solution?

### Page Preview

Support depends on the [wagtail-headless-preview](https://github.com/torchbox/wagtail-headless-preview) package.
This should instead be part of Wagtail core, either as-is implementation or equivalent.

### Images

- Add first-class support for responsive images and multi-format image generation in API responses.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we include focal points?

Some frontends will handle the images. Instead of Wagtail providing various renditions, Wagtail would only supply the original image and leave the resizing to the frontend application. In this scenario it makes sense to include the focal point information in the API response.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep. I see we support accessing the focal point in templates so feels like appropriate feature parity.


### Rich Text

- Add a built-in way to render rich text fields in the API.
thibaudcolas marked this conversation as resolved.
Show resolved Hide resolved
- More reliable [link handling](https://github.com/wagtail/wagtail/pull/9984)

### Password-protected Pages

Missing support in the API.

### Userbar

Only available as a Django Templates template tag or Jinja2 function. Requires:

- Authentication and authorization
- Loading of Wagtail static assets (CSS & JS)
- Loading of Wagtail UI components (HTML / Django Templates)

### Content checks

Only available via the userbar. In addition to the userbar, also requires:

- Page Preview
- Cross-domain cross-frame communication

## Open Questions

TODO
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some questions to consider.

  1. Would we want to consider a more open schema driven response structure ever? We kind of have this with the meta/data response objects but maybe go further with alignment to something like https://jsonapi.org/ or https://www.w3.org/TR/json-ld/ or similar.
  2. What kind of story to we want to tell when it comes to emerging community practice such as HTMX/Turbo style applications? Or do we consider this not really 'headless'?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. While API documentation is nice to have, I don't think it is essential for building a headless site. It shouldn't be something that blocks a consistent approach to headless previews and ensuring the API exposes all data (pages, snippets, redirects), and functions (crud, search, etc.) necessary for a headless client.

  2. To me headless implies my CMS is no longer rendering/serving html. With AHAH (htmx, hotwire, et al) html is still being rendered at the server, so I wouldn't consider it headless. If you had a server responsible for just rendering that was sending hotwire/htmx to the client and not calling on wagtail directly, then that would be headless in my book. I feel like AHAH support should be it's own distinct thing, and it likely to be specific to the AHAH framework in play.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dopry The JSON and OpenAPI specifications are valuable not only for generating documentation but also for generating TypeScript types and even complete client applications. From my experience, using machine-generated types is a must. When the Wagtail data structures change, updating the frontend application is straightforward.

These schemas and types also enhance development with code editor autocompletion, which boosts productivity and reduces the likelihood of errors.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@allcaps it's a productivity boon, but not a requirement to implement a headless site. You can create a headless site without it. In my experience adding and removing properties from frontend data access layers is trivial. Figuring out what properties need to be added/removed/modified from the templates when that happens is the time sink, and schema documentation doesn't resolve that particular problem.