A better method to simulate HID-Keyboard and HID-Mouse for Android device #4034
Comments
|
https://github.com/WuDi-ZhanShen/Android-Gyroscope-MC We can refer to the Android-Gyroscope-MC project on GitHub. The project launches a dex file using the "app_process" command, and then creates a HID device using a JNI function. |
|
Thank you for the suggestion and pointers 👍 This would solve two major problems with HID over AOA:
However, it could not completely replace HID over AOA either: the latter allows to send HID events even when USB debugging is disabled (which is not possible with UHID, since it requires to execute
That's annoying :/ I'd like to keep scrcpy as simple as possible, but adding a native part would require the Android NDK to build, and either we support only x86, either we need to generate .so for multiple arch… However, that would also to support #2130 (which also requires native code). |
I have seen #2130, which mentions the use of the uinput API to simulate input devices, but fails to open "/dev/uinput" on some devices. Indeed, uinput is not available on quite a few Android phones, and I have encountered the same problem and am not sure why. However, uhid should theoretically be supported on any Android phone, as Bluetooth input devices such as controllers, keyboards, mice, and even headphones require registration through uhid. So far, I have not encountered any Android devices that do not support uhid. Creating devices with uinput requires the use of ioctl, and similar operations are also required for uhid. This must be done using native code, as both uinput and uhid need to interact with the Linux kernel. I have also tried to find ways to implement uinput or uhid in pure Java, but without success. |
|
I found Android system already has a JNI wrapper for uhid, I wonder can we use it directly so no native code needs to be build. static {
System.loadLibrary("hidcommand_jni");
}
private static native long nativeOpenDevice(
String name,
int id,
int vid,
int pid,
int bus,
byte[] descriptor,
DeviceCallback callback);
private static native void nativeSendReport(long ptr, byte[] data);
private static native void nativeSendGetFeatureReportReply(long ptr, int id, byte[] data);
private static native void nativeSendSetReportReply(long ptr, int id, boolean success);
private static native void nativeCloseDevice(long ptr);EDIT: It's added in Android 6. |
Maybe we can use reflection to access these JNI functions, although I'm unsure about the performance. I'll give it a try. |
|
I mean maybe you can load |
I'm not sure if we can directly call these JNI functions, since they are limited to being called only by a specific Java class who create these JNI functions, not who load the library. |
|
Sorry I'm not familiar with Java and JNI, can you give your own package and class the same name as the JNI library is looking for, so those native methods get registered on your code (because |
Wow, that's an amazing idea. I'll try. It sounds really feasible. |
https://github.com/WuDi-ZhanShen/AndroidUHidPureJava The method you suggested works fine !! |
Thanks to yume-chan's method, I was able to successfully create uhid mouse and keyboard using pure Java. Please check out my project at: |
Does that mean it's also possible to support game controllers? |
Yes, It does. It also supports hid touchScreen or any hid input devices. |
|
In fact, the native API has changed multiple times, even the // 6
private static native long nativeOpenDevice(String name, int id, int vid, int pid,
byte[] descriptor, MessageQueue queue, DeviceCallback callback);
// 9
private static native long nativeOpenDevice(String name, int id, int vid, int pid,
byte[] descriptor, DeviceCallback callback);
// 11
private static native long nativeOpenDevice(String name, int id, int vid, int pid, int bus,
byte[] descriptor, DeviceCallback callback);I haven't tested, but Google says I can list all overloads in a class, and the native code will register to one of them based on signature. Then I just need to call the correct overload based on system version (or try each one and catch the runtime error). IIRC, HID touchscreen needs to send some feature reports, which is only supported in Android 10 and later. |
I think what Google says is right. And I only test the uhid touchscreen function on Android12 and Android11. |
|
@WuDi-ZhanShen Do you want to create a PR for Scrcpy to add uhid mouse and keyboard support? If not, I will try to do that. My plan is integrating all The The I don't know if there are any advantages of using uhid touchscreen emulation for touch inputs than Android API event injection, so touch events will continue to use API based injection. Game controller support will be added later, with only |
👍 |
Of course, this is mainly your credit. You provided this valuable idea to implement the uhid-pure-java feature, and I just did simple verification. So I think the PR should be created by you. |
I think this is a great feature, I'm hoping to see someone come up with something. |
|
Actually, uhid only needs However, due to an unknown reason, I made a POC using Kotlin on Android end and JavaScript on client end. The Android end is extremely simple (the socket listening part is omitted): import android.system.Os
import android.system.OsConstants
import java.io.InputStream
import java.io.OutputStream
import kotlin.concurrent.thread
class UHid {
companion object {
private const val MESSAGE_OPEN = 0;
private const val MESSAGE_WRITE = 1;
private const val MESSAGE_ERROR = 2;
private const val MESSAGE_READ = 3;
private const val MESSAGE_CLOSE = 4;
private fun writeInt32(writer: OutputStream, value: Int) {
writer.write(value)
writer.write(value shr 8)
writer.write(value shr 16)
writer.write(value shr 24)
}
private fun readInt32(reader: InputStream): Int {
return reader.read() or (reader.read() shl 8) or (reader.read() shl 16) or (reader.read() shl 24)
}
fun openUHid(input: InputStream, output: OutputStream) {
val fd = Os.open("/dev/uhid", OsConstants.O_RDWR, 0)
writeInt32(output, MESSAGE_OPEN)
thread {
try {
while (true) {
when (readInt32(input)) {
MESSAGE_WRITE -> {
val size = readInt32(input)
val data = ByteArray(size)
input.read(data)
Os.write(fd, data, 0, data.size)
}
MESSAGE_CLOSE -> {
Os.close(fd)
break
}
}
}
} catch (e: Throwable) {
e.printStackTrace()
try {
Os.close(fd)
} catch (e: Throwable) {
// Ignore
}
try {
writeInt32(output, MESSAGE_ERROR)
} catch (e: Throwable) {
// Ignore
}
}
}
val buffer = ByteArray(8 * 1024)
try {
while (true) {
val read = Os.read(fd, buffer, 0, buffer.size)
writeInt32(output, MESSAGE_READ)
writeInt32(output, read)
output.write(buffer, 0, read)
}
} catch (e: Throwable) {
e.printStackTrace()
try {
Os.close(fd)
} catch (e: Throwable) {
// Ignore
}
try {
writeInt32(output, MESSAGE_ERROR)
} catch (e: Throwable) {
// Ignore
}
}
}
}
}The client end needs to implement uhid protocol, and emulate HID devices. I have successfully created a keyboard and a gamepad. Because uhid supports output reports, the keyboard can detect capslock state on Android side (can be used to support syncing capslock status), and gamepad rumble also works (on Android 12+). |
|
Awesome 👍 Maybe the next "big" subject (after audio, camera, turn_screen_off on Android 14…). Or maybe #4380… Thank you for all your research work ❤️ |
|
You could also spawn the |
Hi! |
The client side is in an internal project. You can refer the on-going PR #4473 |
|
Superseded y #4473 I guess, released in scrcpy 2.4. |
https://github.com/torvalds/linux/blob/master/samples/uhid/uhid-example.c
We can utilize the Linux uhid API to create a HID input device, and having ADB permission is sufficient to achieve this.
This method doesn't require an actual USB wire to be connected to the Android device, and allows the HID keyboard and mouse functionality to coexist with other features of scrcpy.
However, this method requires scrcpy to load an ELF file, as the uhid API must be called in JNI.
The text was updated successfully, but these errors were encountered: