Building UI components into Excalibur

[cdelstad]

I'd like to get some thoughts out to start a discussion regarding creating UI components built into Excalibur that meets accessibility as well as web standards.
I haven't thought everything all the way through, just wanting to get the conversation started.

General:

1. CSS Variables should be used so component styles can more easily be overridden.
2. The components should be white label - which means no styling other than fundamental functionality and accessibility. Buttons having display:inline-block or display:block for full width, would be an example of fundamental functionality. Accessibility would be that buttons would have a hit box a minimum of 44x44px. ING Lion web components does a good job of implementing accessibility patterns, though sometimes a bit more complex than I feel is needed for modern browsers. It seems logical that they needed to support older browsers.
3. If we go the route of absolute positioning, we can use engine.screen.worldToPageCoordinates(...) though we will need to have resize functionality to move things around.
4. Semantic HTML is king vs. over-depending on aria-label, etc.
5. Follow WCAG 2.2 accessibility guidelines.
6. UI components should be able to be tested with browser tools like WAVE and aXe, and also via unit tests with a library such as axe-validations.

An overlay is not necessarily required as putting HTML elements above the canvas would work. Though I do think it helps manage the UIs as a whole.

Overlay/ Container:

1. An overlay or container will be created with its own root element to contain all the UI's. It will sit on top of the canvas, with a high zIndex.
2. It should be developed as such so we can easily disable either the overlay/container and its elements, or can disable the game/canvas. This will allow us to keep focus on this layer to avoid focus traps, and the like, and help make the UI's more keyboard accessible.
3. Interactions such as clicks, pointers, touch, etc. need to be able to pass through the overlay/container to the canvas.
4. The overlay/container will default to the same size and coordinates as the canvas, but can be overridden - should someone want to be able to have the controls in a location other than directly over the canvas.

HTML Components:

1. Create common components such as button, text, statusBar (used for HP, Mana, and progress bars), Dialog box, Region (more of a wrapper around a section of the UI to provide an ARIA region for Screen Readers to be able to quickly navigate to that section).

These examples are me trying to visualize and give us more concrete examples for discussion.

this.myText = new Text({
id?,
element: 'div', // or span, p, etc.
pos?: vec(10,10), // would put the text box at 10px, 10px
alignment?: 'vertical-center horizontal-center' // would position in the center of the canvas
text: 'this is some text',
classes: 'font-h2, red'
aria-label?,
disabled?
onClick?: () => this.changeColor()
});

// another pattern
this.myText = new Text('myId', 'div', 10, 10, 'this is some text', {font-h2, red}, () => this.changeColor());

I am not a big fan of the second pattern. It is an older approach. I feel it increases cognitive load for the developer and anyone reading their code because you need to remember the parameter list, or need to break away from what you are doing to look it up. And then there are the nulls that need to be in place when you want to use a parameter later in the list.

2. A Generic HTML component that gives developers something like an escape hatch. You would pass in an html template literal and it would be rendered as is. The developer would be responsible for all wiring and interaction. It would need a DOM element to know where to put it, as well as a parameter for append, prepend, etc. so the components know where and how to add it to the DOM.
   We would need to think about how we could implement reactivity for the generic HTML component. Perhaps a generic event listener? It should also allow for variables/placeholders for values?

this.myHTML = new HTML({
root: 'right-here',
insertMethod: 'append', // also prepend, before, and after?
html:`<h1 class='font-h1 red'>${helloWorldText}</h1>
	  <button class='primary'>Click me!</button>`;
});

Maybe even a combined approach with 1 and 2:

this.myHTML = new HTML({
root: 'right-here',
insertMethod: 'append', // also prepend, before, and after?
html:`<div region='heads up display'>
	    ${this.myText}
	  </div>
});

3. Reactivity: How to update the UI component from game data? We could UI events into the EventEmitter API to pass down, but how would we update the UI component? I suppose with a reference to the UI element we can update it like changing text, or statusbar level, etc? I am trying to stay more vanilla HTML, though Lit web components has a lot of nice reactivity built in and if we were going to need to write a whole reactivity system, it's going to be tough to beat Lit's 6kb size. ;) We could build a simple state management system, basic ones can be written in ~100 lines of code, though it won't handle deep objects and would register a change with any part has changed, even if it is not pertinent to what you need.
   Note: In Phaser3, you set a button as interactive and that wires up an EventEmitter.

[kamranayub]

One of my thoughts has always been whether we could actually use JSX for UI with Excalibur. JSX is agnostic of framework, but it provides the familiar XML-like DOM syntax we could then transform into Excalibur logic (like ECS). It's composable, which fits with the general idea in Excalibur of having primitives that you can build into more complex things. That way, we could provide a set of basic, accessible UI building blocks that people can build off of, or even create whole design systems on top of as separate projects.

The goals here would be not to necessarily tie Excalibur straight to a DOM UI implementation; but to use JSX as the abstraction so that we can support a flexible renderer system. This would allow you to extend, or other people to extend, so you could build custom renderers.

In this way you could do stuff like this:

class Menu extends Scene {

  public drawUI(engine: Engine) {
    return (
      <MainMenuList pos={vec(engine.halfDrawWidth, engine.halfDrawHeight)}>
        <MainMenuButton to={LevelOneScene} />
        <MainMenuButton to={SettingsScene} />
     </MainMenuList>
    )
  }
}

This way you could express the UI tree in a familiar way instead of new MainMenuList(...) etc. Engine could then provide a way to switch the renderer.

It's only a thought though, I don't know what this really entails 😅

Prior Art

• I found this OneJS which is something similar for Unity
• Ink uses JSX to build CLIs

[andrico1234]

I think integrating a way to build UIs on top of Excalibur is a great idea. I still don't quite follow the exact problem you're trying to solve though. Is it:

1. Offering a way for user-authored components to hook into the engine state
2. Offering a syntax that allows developers to write excalibur components
3. Offering consumers a way to build components that adhere to the “Excalibur” style guide

The solutions for each of these sound like 3 different levels of abstraction:

1. Low-level primitives that are compatible with one (or hopefully more) different flavours of writing markup, that the user will write themselves (agnostic of Excalibur)
2. Providing the users with an Excalibur flavour of writing markup that hooks into the engine state by default. (e.g., your new HTML() markup language).
3. Building ready-made components that adhere to an Excalibur style-guide (or headless).

I’m personally leaning to option 1 for many reasons. There are already a lot of ways to write markup, and I worry that requiring people to use a different syntax for writing UIs might increase friction. It's also probably going to take a lot of work to get the DX feeling right. As a web developer, I'd feel right at home building UIs in my template language of choice (either Svelte or Lit).

Off the top of my head, there are a couple of ways that we can achieve this:

1. either an event system that bubbles up past the canvas element that components can listen to.
2. State management tool like Nanostores. Nano is used by Astro (a JS metaframework that allows developers to drop in React, Svelte, Vue code in the same codebase and share state using Nanostores).

I haven’t used Nanostores much, but it’s a framework-agnostic way of sharing state. Maybe it could be used to share the state between the canvas and any UIs on top of it.

I think this would be a good start because building a suite of accessible components that are battle-tested is hard. In fact, there are a lot of tools that offer headless components, that just deal with the accessible implementation and then let developers style it to their heart’s content. Radix UI, MUI Base, Shoelace. The problem with this approach is that you may be locking users into a specific ecosystem (Radix and MUI are built on React).

Shoelace is built on top of Web Components which is supported natively in the browser, so that might be something to consider.

Many tools like React Aria have been in development for years, and are dedicated to the cause of building accessible components. I think it would be tricky to try and build something of that scale from scratch.

My biggest concern with option 3 is the idea of providing ready-made components that are both specific to video game UI, while being flexible enough to allow developers to build anything they want with it.

I haven’t used any other game engines, and so I don’t know how they approach drop-in UIs, so I don’t have as much to say about this.

[cdelstad]

@kamranayub This looks wonderful. Nice work. I love composability.
I was trying to stay with a vanilla solution to reduce learning curve and complexity. I see a lot of folks in game dev groups wanting to develop a game that ask what they need to learn to be able to develop a game. Often they aren't very familiar with JS, HTML, nor CSS. JSX is one more layer to the learning curve and is not the most intuitive, at first, as it makes up its own rules that are not consistent with HTML. ;) Now, this is not a reason not to use JSX, as there are benefits to using it, it is something we should include in our considerations.
I am thinking the higher the learning curves can cause more friction. That causes folks to bounce and try something else. Perhaps a blend similar to my suggestions. Pre-built components folks can use easily, that may use JSX under the hood? Similar to my HTML approach, folks can write vanilla HTML or JSX in the JSX format instead of template literals.
I have concerns with the canvas UI option. Solutions in the canvas are horrible for accessibility, and folks may default to it as the most simple solution, even if it's just perception. TBH, I'm not crazy about giving users an option that doesn't help with accessibility.
Just spitballing the effort levels in my mind and the renderers solution will be a lot more work, especially for testing and maintenance concerns. That being said, I do like being able to wire in third party solutions.
@andrico1234 Shoelace and other web component libs use Google Lit and I am a big fan of this approach. Heck, even Adobe Photoshop on the web uses Lit web components. ;) As far as building a component library that is accessible, it isn't as daunting of a task as it seems. It's hard at first, but as you get going, it's repeating a lot of the same patterns over and over. Honestly, just following semantic HTML gtakes you a long way. ie: make a button a button, not a div.

[andrico1234]

(Funny you should say that, I'm currently building a course all about building a web component library using Lit).

As for building accessible components, I agree that building a strong foundation can get you most of the way there. It's only with complex patterns, like notifications, and select menus where things can get pretty dicey.

[cdelstad]

Very cool! Yes, some components can get pretty dicey. We had to make a carousel work for accessibility on mobile devices. We use the Announcer pattern which sets up multiple aria live regions that our code alternates writing to and cleaning out with a X milisecond delay - just long enough for the screen reader to pick it up to read. It works very well, though you always need to be careful about how many assertive or alerts are sending because they will clobber each other.

[cdelstad]

Another brainstorming thought that came to mind in thinking about using Lit with React. There is lit-react lib which adds a wrapper around a Lit component that wires it into the React lifecycles.
We could do an Excalibur wrapper that can wire any UI component into Excalibur ecosystem and lifecycle methods. IE: You have react components, this wrapper will wire their reactivity into Excalibur events, lifecycle methods, etc. no we can't be a specific as lit-react because I don't think we would want to need the write wrappers for every UI framework.
So this would mostly be a way to wire in Excalibur ecosystem into HTML and expose ot there for use. It would be the responsibility of the developer to actually wire the items exposed to their HTML and/or components.
On the other hand a state management system does the same thing and UI libraries already have reactivity with that.
Anyway, something to add to our things to consider.