Define a low latency event that isn't occur in the document lifecycle

[dtapuska]

Chrome has experimented in shipping events relative to rAF. pointermove is only sent once for every rAF callback and occurs just before the rAF callback. Specifically this was done for touch in Chrome 59 and mouse in Chrome 60. See some discussion here #9.

With pointerevent dispatched at rAF time it is useful for some apps that don't produce any in the normal document lifecycle (see issue whatwg/html#2659) to have input right away.

I was wondering if it we should add a pointermove-raw event to enable the cases when pointer moves with low latency are needed. We want to ensure that this isn't the typical usage as the raw events can occur at a much higher frequency (eg. 1000Hz) so I don't think we want to use the same event type as that would conflate the problem.

Talking with @RByers we discussed that this event shouldn't bubble and is non-cancelable.

[dtapuska]

@patrickkettner @smaug---- @scottgonzalez Any thoughts on this?

[scottgonzalez]

Is there a reason that the much higher frequency is beneficial as opposed to just dispatching at a predefined rate closer to the desired rAF frequency, such as 60Hz? If not, then we could just continue to use the existing events.

[smaug----]

I think we should try to use the same events, but possibly let the web page to hint that it wants high precision events.

[dtapuska]

It is all about latency as you might need to wait like 16ms for a rAF call to come around. This is a very limited use case but has the potential to deliver a good user experience for apps that use it correctly.

That was my initial approach where I specified an event listener option.

@RByers said the problem with that approach is that it doesn't allow apps to reason about performance because it can be modified in a number of ways. For example if an iframe sets the option then all event listeners even 3rd party code on the main page (for usage metrics) starts getting executed more frequently where they only expected it at rAF time.

[scottgonzalez]

But what are the events being used for? Why does drawing to an offscreen canvas faster than rAF improve the user experience? I assume the rendered canvas is then being injected into a visible location; but that should be done inside rAF, right?

[dtapuska]

We are working on having an on-screen canvas actually get a hardware layer (like video). Chrome has like a multi-frame swap so you can avoid the frame swaps with the direct hardware layer. So you can get an onscreen canvas swapping in a single frame.

[RByers]

Here's another use case that (while less compelling) is a lot simpler and easier to reason about IMHO: a music synthesizer / DJ application that creates/modifies sound somehow in response to touch position. In that case high frequency and low latency can be very important, while aligning with any form of visual output is unimportant (though there's probably still some visual feedback somewhere, the primary output is audio).

The problem I have with a global "use high frequency" hint is that whether or not you want (and are designed to handle) high-frequency events is really a per-handler question. We know from our touch scrolling work that the vast majority of touch event listeners installed on the web today are for non-primary use cases (eg. activity monitoring, analytics, etc.). If we change the timing of an event for one handler then invariants expected by other handlers for the event (both around timing and frequency) would be broken. The cross-iframe case here is the worst since there's not necessarily any chance for the impacted code to co-ordinate, but the problem exists within a single frame as well.

If we don't want to introduce a new "pointerrawmove" event type then other options I can imagine that avoid the above problem are:

1. A "enable unbuffered mode" API (or option to setPointerCapture) that can be called on pointerdown. This most directly mirrors the Android API. To avoid the above issues I'd ideally want calling this API to disable event bubbling, so global activity monitor listeners would just not see the movement at all rather than force coalescing and loss of frequency for the high-frequency handler, but maybe it would be sufficient to encourage consumers of this API to call stopImmediatePropagation? I guess that would likely encourage these niche applications to use a capturing listener on the window to try to get priority on the event stream.

2. Add a polling API (like used by Gamepad) where an app can query, at it's own rate, for the current state of a pointer. This has a lot of overlap with prediction - Consider adding API to get predicted points #38, I could imagine an API like PointerEvent#estimatePointerState(DOMHighResTimestamp) which returns a PointerEvent with the estimated position (and other properties) for the input pointer at the given timestamp. Or maybe the provided timestamp is just a hint and the application should rely on the timestamp in the returned event, eg. so we can refuse to try to predict farther in the future than is reasonably practical, or even refuse to predict into the future at all if not supported by an implementation. One of the things I like about this proposal is that it would let us eliminate DOM overhead (such as event path computation) entirely such that maybe processing at 1000hz is not at all unreasonable even on a phone.

WDYT?

[Rycochet]

I'd add a third suggestion to @RByers list there -

3. Add an Observer API for the raw events as they come in. Make use of the high precision performance.now() timestamp within it - that way the observer has the exact time it came in synchronised with the rAF clock. Apologies for the TypeScript, but easier to show structure:

interface InputObserverInit {
   mouse: boolean;
   touch: boolean;
}

interface InputRecord {
   type: "mouse" | "touch";
   pageX: number;
   pageY: number;
   which: number;
   // etc - exact contents dependant on type, and similar to MouseEvent / TouchEvent
}

class InputObserver {
   constructor(callback: (record?: InputRecord, instance?: InputObserver) => void);
   observe(init: InputObserverInit);
   disconnect();
}

...One potential benefit of this would be the ability to add this to all user input events including keyboard and device orientation etc.

Games are the obvious benefit for real-time input. Even though the display is clamped at rAF speeds, any form of twitch gaming wants to be able to react to things when they happen. As above, the display is just an indication of the state, but the state itself doesn't conform to 60/30/15fps etc. Especially on slower devices, or heavier games that end up pushing the framerate down, locking the input rate to the display rate will limit what is possible in an inconsistent way.

[dtapuska]

@Rycochet event.timestamp is relative to the Performance.now() in Chrome.

This is an interesting idea. It definitely gets rid of the problems with hit testing that would dominate a high frequency mechanism.

[smaug----]

But ends up copying lots of event handling mechanism. And hit-testing is rather useful thing to have, since that gives you also proper event propagation.
And hit testing doens't need to be slow for example when events are dispatched all the time to a .

[Rycochet]

It's always possible to get the element under the pointer via window.elementFromPoint (which could be caching if it's the current mouse pointer position on a per-mouse-move bases) - if someone is wanting a really high speed from the mouse then it's more likely that it'll be using WebGL or Canvas, or interested in the delta rather than the element under it (which is unlikely to change much between rAF based mouseover / mouseenter events).

The only things that are needed from a MouseEvent / TouchEvent are the position and type - for the mouse it's pageX / pageY / which, but touch has changedTouches with a little more detail on it. If gamepads also got added (as an addition to the Gamepad API) then it would likely have yet more attributes (but that would be subject to a different discussion) ;-)

[smaug----]

Curious, why pageX/Y? I would assume clientX/Y.

Event propagation let's one to add event listeners to different places and using normal event handling mechanisms. Inventing something totally new is possibly a bit heavy (and complicates the already complicated platform even more).

But I don't have a strong opinion yet. I'd prefer to try to reuse the existing mechanisms if possible.

[Rycochet]

pageX / pageY are just the ones I tend to use the most - clientX and clientY are just as useful - and both can be used happily enough. Thinking about it I'd be almost convinced to keep this as simple and fast as possible and just use screenX / screenY and possibly add a deltaX and deltaY. I think what coordinates are included would depend on the most likely uses for it...

Observers aren't new - they're used for MutationObserver among other things (PerformanceObserver, IntersectionObserver, FetchObserver etc) - specifically used as a replacement for MutationEvent because it needed something more performant than an event with bubble and cancel.

[smaug----]

Limiting the raw data to capturing mode only feels like a hack.
(#260 (comment))
That could force a web page to use both types of pointer move events.
What is the reason to not have an API to let web page to hint whether to coalesce or not? It could be per browsing context.

[NavidZ]

The use case of listening to both types of pointermove and pointerrawmove is quite valid. They serve different purposes.

The argument against having a "browser context"-wide api to enable/disable rAF-aligning is the use case of using other third-party libraries or just a big webpage which has a lot of components. With disabling rAF for the whole browsing context you are forcing all the other handlers and their bubbling handlers and everything to be run with a higher frequency even though they might not care about that. Introducing a new event easily lets us make it so that it is isolated from other handlers and also we can make cancelable and bubble of that to false to make sure it is not causing any additional processing. The bubble part of it is debatable as if the parent has a pointerrawmove handler it has already expressed its interest in processing those events.

Regarding limiting it to the capturing mode I don't have a strong argument for it. I'm very open to the possibility of having it fired all the time if there are handlers on the page. If the page desires to circumvent the browser hit-testing (for performance reasons or whatever else) they can always use the capture api.

[NavidZ]

@smaug---- does this address your concerns? More explicitly not limiting the pointerrawmove to the capturing and also set the bubble to true.

[smaug----]

@spanicker I think I mentioned this spec issue to you at some point, but if not here it is.
This may affect to how UAs align (or not align) input events to rAF.