Relay touchscreen / pen input from Android screen back to PC (turn Android device into a pen tablet for PC)

[ilyesgouta]

• I have checked that a similar feature request does not already exist.

Is your feature request related to a problem? Please describe.
The Android device (phone or tablet) screen could act as a pen or digitizing/display tablet (e.g. see wacom cintiq or intuos) to capture pen input (moves, press/release, strokes) to aid in drawing or sketching activities on the PC.

Describe the solution you'd like
Scrcpy to forward input events captured on the Android device to the Windows / Linux / Mac host. Input events here are gestures and interactions captured from the Android device touch screen. Ideally the solution wouldn't incur significant latency between the capture of the event and its restitution on the host computer. E.g. for a minimal latency implementation, a virtual Pen driver could be implemented for Windows host, so that input events from the Android device could become available to all user apps running on the Windows host (and uniformly handled by the Win32 API).

Describe alternatives you've considered
Other solutions include using a client/server approach, which could also work by injecting captured input from ADB device into the host PC (Win/Linux/Mac).

Additional context
Having the feature implemented in Scrcpy would be wonderful, as Scrcpy is an open source program, accessible to everyone on all major platforms.

[ilyesgouta]

Windows Digitizer Class Input Devices (drivers) are documented here: https://docs.microsoft.com/en-us/windows-hardware/design/component-guidelines/windows-digitizer-class-input-devices

[ilyesgouta]

E.g. for the case of Windows, Scrcpy could be loading a new Pen HID device driver (bundled with Scrcpy package), passing input captured from the Android screen / device and received from ADB, to the new HID driver.

Because it is a standard Windows device driver dispatching input, applications like Inkscape, Adobe Illustrator, Microsoft Whiteboard, could then leverage these new input events.

[rom1v]

Hi,

IMO, this is out of scope for scrcpy, which allows to control the device from the computer (not the other way around).

Technically, it's also quite different (capturing device events, no video stream, injecting system-specific events to the computer, etc.), so there would be no benefit to implement it in scrcpy rather than a totally separate app.

[ilyesgouta]

IMO, this is out of scope for scrcpy, which allows to control the device from the computer (not the other way around).

Technically, it's also quite different (capturing device events, no video stream, injecting system-specific events to the computer, etc.), so there would be no benefit to implement it in scrcpy rather than a totally separate app.

It is a very useful feature and Scrcpy has all the foundation in place to implement this as an additional extension. Please consider adding it.

Windows Pen protocol implementation is documented here: https://docs.microsoft.com/en-us/windows-hardware/design/component-guidelines/pen-protocol-implementation, and likely an implementation as a 3rd-party HID mini-port driver would be needed, that is to translate the received events from Android to the Windows' HID device class.

[ilyesgouta]

E.g., using the getevent tool available on most Android devices, https://source.android.com/devices/input/getevent, can be executed on the Android target by Scrcpy

$ adb shell getevent -lt /dev/input/event1

and events retrieved by ADB then translated to native events, e.g. on Windows platforms, using a new Pen HID mini-driver or using SendInput() Win32 API (could be easier).

[ilyesgouta]

@matulek adb shell getevent -lt /dev/input/event1 actually gets you all the needed information. scrcpy has to launch that command through adb, redirect stdout using pipes and read / parse the incoming data (which is already formatted, e.g. lines have the same length) and then just use the Win32 SendInput() API to (re)synthesize the mouse events on Windows.

I really hope this gets implemented in scrcpy.

[matulek]

@ilyesgouta after fiddling a little with scrcpy I realized that it doesn't allow to use android as the external monitor, so, after all, this feature unfortunately seems out of scope to me.