[bug] Lots of internal_on_mousemove IPC calls causing lag and cursor flashing in 2.0.0-beta.0

[Unit2795]

Describe the bug

After migrating from Tauri V1 to V2 beta, when I move my mouse cursor over a button, the pointer "hand" symbol flashes between a normal cursor and the hand. A lot of traffic is observed in the Network tab of the devtools directed at this endpoint (http://ipc.localhost/plugin%3Awindow%7Cinternal_on_mousemove). Removing "window:default" in "/src-tauri/capabilities/mycapabilities.json" stops the flashing but then a lot of errors are logged in the console. I attempted to make a project from scratch and noted the same issue.

Attempted Fixes:

• Remove "window:default"
• Update Node, pnpm, Rust, Windows, package.json, cargo.toml

---

Cursor flash on hover/move with "window:default" enabled

2024-02-04_23-39-27.mp4

Console errors with "window:default" disabled

2024-02-05_00-37-58.mp4

Reproduction

1. Run pnpm create tauri-app --alpha
   1. Use any project name
   2. Select Typescript/Javascript
   3. Select any package manager
   4. Select any UI template
   5. Select yes or no for setting project up for mobile
2. Run npm install in project directory
3. Run npm run tauri dev
4. Once application has started, move mouse around in application window and hover over a button, open inspector and make note of console logs and/or network.

Expected behavior

1. Cursor should not flash between normal cursor and hand pointer when hovering over buttons
2. Excess events for mouse movement should not be fired, if possible (or be able to opt into this)

Full tauri info output

[✔] Environment
    - OS: Windows 10.0.22000 X64
    ✔ WebView2: 121.0.2277.98
    ✔ MSVC: Visual Studio Build Tools 2022
    ✔ rustc: 1.75.0 (82e1608df 2023-12-21)
    ✔ cargo: 1.75.0 (1d8b05cdd 2023-11-20)
    ✔ rustup: 1.26.0 (5af9b9484 2023-04-05)
    ✔ Rust toolchain: stable-x86_64-pc-windows-msvc (default)
    - node: 20.11.0
    - pnpm: 8.15.1
    - npm: 10.4.0

[-] Packages
    - tauri [RUST]: 2.0.0-beta.2
    - tauri-build [RUST]: 2.0.0-beta.1
    - wry [RUST]: 0.35.2
    - tao [RUST]: 0.25.0
    - @tauri-apps/api [NPM]: 2.0.0-beta.0
    - @tauri-apps/cli [NPM]: 2.0.0-beta.0

[-] App
    - build-type: bundle
    - CSP: unset
    - frontendDist: ../dist
    - devUrl: http://localhost:1420/
    - framework: SolidJS
    - bundler: Vite

Stack trace

POST http://ipc.localhost/plugin%3Awindow%7Cinternal_on_mousemove 400 (Bad Request)
value @ VM10:70
(anonymous) @ VM12:130
action @ VM12:269
(anonymous) @ VM12:278
value @ VM12:252
(anonymous) @ VM12:400
VM12:263  

Uncaught (in promise) "window.internal_on_mousemove not allowed. Permissions associated with this command: window:allow-internal-on-mousemove, window:default"
(anonymous) @ VM12:263
value @ VM12:227
(anonymous) @ VM10:96
Promise.then (async)
value @ VM10:94
(anonymous) @ VM12:130
action @ VM12:269
(anonymous) @ VM12:278
value @ VM12:252
(anonymous) @ VM12:400
VM10:70

Additional context

• Just upgraded from Tauri V1 to V2 beta

[FabianLars]

Probably should have included the error in #8750 lol. but yes, we agree that it's too hard on the ipc and will look into it. (that said, i don't see the cursor flashing)

Edit: I closed my own issue cause yours is better, for visibility here's my comment from the other issue:

This is mostly just a reminder issue because i have to log off for today.

The resizing implementation we introduced in #8537 is pretty hard on the ipc, even if the window uses default decorations. I only noticed this because i was missing the default window permissions so i had thousands of error messages in the devtools console complaining about a forbidden internal command (which tbh is also weird).

Ideally the mousemove event could be handled in rust only but i'd imagine there's a reason we didn't do that so maybe the best we can do is to disable the listener in undecorated windows?

[amrbashir]

For transparency, the solution in #8537 was already used before in wry so the performance should be the same and because webview2 doesn't provide anyway to hit test the webview from native side, however on Linux, we used to use raw GTK APIs and hit test on the native side, I will see if I can bring that back.

[FabianLars]

Something i didn't think about, this also seems to trigger #8177

[Unit2795]

I haven't contributed to Tauri, I may be mistaken but it appears the code this is involved in is used to:

1. Resize the window when undecorated. As set in tauri.conf.json app.windows[x].decorations property.
2. When you mouse over a region on the edge of the window, the cursor is changed to the proper resize cursor for its location.
3. If you click in this region and then drag, the window is resized.

What I'm wondering is:

1. Shouldn't this behavior be disabled when the window IS decorated with a config check in the code if possible, since it seems unnecessary in this case (or be explicitly opt-in if you need it for some reason)?
2. Could we add a region around the window in HTML with a mousedown & mousemove event so that these only fire when in this region? Or have a conditional where JS only sends these IPC calls when the mouse is over the edge regions?

Just ideas, may not be applicable or feasible. If you'd like for me to take a crack at it, I'd be happy to make my first contribution and open a branch/PR. Let me know if there is anything special I should do to get started. Otherwise I'll just check the contribution guide.

Please advise.

Thanks.

[Unit2795]

Here is a close up of the flashing cursor I noticed (I am not clicking or anything, just moving the mouse around over the button)

2024-02-06_09-29-46.mp4

[Pukepaiy]

the same issue,and the right and bottom cannot be resized

[amrbashir]

Here is a close up of the flashing cursor I noticed (I am not clicking or anything, just moving the mouse around over the button)

2024-02-06_09-29-46.mp4

Nice catch, I will fix that as well in the PR

[neil4dong]

Hi, I look little deeper in to Tauri. tauri-2.0.0-beta.2\src\window\scripts\undecorated-resizing.js

Not understand the reason why
osName !== 'macos' && osName !== 'ios' && osName !== 'android'
should send a mousemove event to ipc. This code barely equals to :on windows platform send a mousemove event.

And I also checked the backend code. \tauri-2.0.0-beta.2\src\ipc\authority.rs
the backend response to the front that command not allowed. because the command not in the allowed_commands

Is this a in progress feature or not?
If it is windows platform only feature, perhaps we should add to the allowed_commands

[Unit2795]

Amr created a PR that is awaiting approval that fixes these issues. The OS check and permissions issue has been resolved. The OS check is now performed in Rust and checks for Windows alone, no script is attached for non-Windows platforms. And the permissions issue is avoided by using a custom invoke system (window.ipc.postMessage) based on postMessage/eval that avoids the permissions & runtime authority.

[neil4dong]

The implementation of this is quite hevavy for just to change the cursor style and check able to resize. MouseMove trigger very frequently. maybe need a debounce

[Unit2795]

Yeah, it's not ideal. From what I've heard I don't think this will end up being the final solution. Debouncing is a good idea (200ms or so?). I know Amr is looking into further efficiency improvements. I've been experimenting with possibly getting rid of the JS code entirely or involving some conditionals and fancy CSS.

[Unit2795]

Just from what I've seen so far; I'm not sure why this couldn't be handled entirely natively in Rust using the Win32 API from the windows crate. It seems like this probably needs to be addressed within Tao. Undecorated windows in Tao on Windows don't operate as I'd expect them to. Going to look at it more but I suspect something may have came up that made this difficult. Going to snoop around and try some things in Tao.

[amrbashir]

tao does support resizing undecorated windows natively, however when adding a wry webview, which is basically WebView2 on Windows, it is not a simple widget that is added to the tao window, it creates a new child HWND that fills the tao window and so any events related to mouse clicks will be on their HWND and won't reach to the underlying tao window.

[Unit2795]

Ah I see now. It appears WebView2 can't be subclassed either so intercepting hit tests wont be possible (though I notice a tiny 1 pixel region at the top of an undecorated window shows a resizing handle on hover as it's not part of the client area). Looks like JS probably is the best solution.

Small possible remaining improvements (I might give them a go but should not hold up the current PR):

1. Handle changing cursors entirely in JS, no IPC
   1. Will need to fetch decorated, resizable, maximization status from Rust and know when this changes by evaluating a script on the page.
   2. Get DPI and set border region size in pixels in JS (what if the window moves between monitors or spans two monitors with differing DPI?)
   3. When within border region, set cursor in JS using CSS targeting html element (User CSS might interfere with this?)
   4. Debounce the cursor change to reduce the number of calculations
2. Try adding resizing borders to the HTML around the window with corresponding CSS for cursors
   1. The user could possibly remove these if they clear out the document body or set a z-index that is higher than these borders, might be too flimsy. Would need to observe these elements and add them back if they are removed.
   2. Attach events directly to these HTML elements if possible.
3. Only attach the mousemove event for sending resizing IPC calls when the left click mousedown event is fired, and remove it after mouseup.

[amrbashir]

Small possible remaining improvements (I might give them a go but should not hold up the current PR):

1. Handle changing cursors entirely in JS, no IPC

   1. Will need to fetch decorated, resizable, maximization status from Rust and know when this changes by evaluating a script on the page.

   2. Get DPI and set border region size in pixels in JS (what if the window moves between monitors or spans two monitors with differing DPI?)

The reason why I didn't go with this solution before, is that reloading the webview will make it lose the stored state or get out of sync. Maybe a mix of IPC and stored state could work.

3. When within border region, set cursor in JS using CSS targeting html element (User CSS might interfere with this?)

Did test this idea before and it worked pretty well, however we need to use weird class names so as to not conflict with user (i.e. __TAURI_INTERNAL_CLASS_nw-resize__)

2. Try adding resizing borders to the HTML around the window with corresponding CSS for cursors

   1. The user could possibly remove these if they clear out the document body or set a z-index that is higher than these borders, might be too flimsy. Would need to observe these elements and add them back if they are removed.
   2. Attach events directly to these HTML elements if possible.

I don't agree with this idea at all, since it will interfere with the user DOM hierarchy and could lead to hidden behaviors that they can't explain.

3. Only attach the mousemove event for sending resizing IPC calls when the left click mousedown event is fired, and remove it after mouseup.

This will prevent the cursor from changing when on the window edge and users won't know if they are on a resizable edge or not.

[giddyos]

I ran into this issue as well and it basically made tauri unusable for me. I wrote up a quick workaround for the odd mouse behavior using the isolation pattern.

When my app starts (svelte in this case) I get the window size and wait for the iframe __tauri_isolation__ to load. Once loaded I send it the initial size and then subscribe to tauri://resize event and send it the updated size onwards.

import { Window } from "@tauri-apps/api/window";
import { onMount } from "svelte";
  
const win = Window.getCurrent();
let iframe: HTMLIFrameElement;

async function updateDimensions() {
    if (!iframe?.contentWindow) return;

    const size = await win.innerSize();

    iframe.contentWindow?.postMessage(
      {
        type: "resize",
        width: size.width,
        height: size.height,
        is_fullscreen: await win.isMaximized(),
      },
      "*"
    );
  }
 
onMount(async () => {
    while (!iframe?.contentWindow) {
      iframe = document.getElementById(
        "__tauri_isolation__"
      ) as HTMLIFrameElement;
      await new Promise((resolve) => setTimeout(resolve, 100));
    }

    await updateDimensions();
});

win.listen("tauri://resize", async () => {
  await updateDimensions();
});

This is what I do in the isolation app:

let width = 0;
let height = 0;
let is_fullscreen = false;

window.addEventListener("message", function (event) {
  // maybe the origin should be checked and some other stuff idk

  if (event.data.type === "resize") {
    width = event.data.width;
    height = event.data.height;
    is_fullscreen = event.data.is_fullscreen;
  }
});

window.__TAURI_ISOLATION_HOOK__ = (payload) => {
  if (payload.cmd === "plugin:window|internal_on_mousemove") {
    let x = payload.payload.x;
    let y = payload.payload.y;

    let min_x = 10;
    let min_y = 30;

    // console.log(`x: ${x}, y: ${y}, width: ${width}, height: ${height}`);

    if (is_fullscreen || !(y < min_y || y > height - min_x || x < min_x || x > width - min_x)) {
      payload.cmd = "nothing";
      payload.payload = {};
    }
  }

  return payload;
};

It still spams ipc calls which isn't ideal and I'm waiting for a fix but this fixes the stuttering issue and be sure to make a tauri command that doesn't do anything so that the console isn't spammed with errors about the endpoint not existing.

Edit: I don't see why this behavior is needed if the app is decorated so in the code i sent above you can use win.isDecorated() and override the event(s) related to window resizing

[Raduc4]

Encountered the same issue but without cursor blinking.
beta.1
Ubuntu 22.04