Document UHID

Rework the documentation to present the keyboard and mouse input modes.

PR #4473 <https://github.com/Genymobile/scrcpy/pull/4473>
This commit is contained in:
Romain Vimont 2024-02-29 10:00:56 +01:00
parent dd479ed176
commit b9d244b4c9
8 changed files with 262 additions and 169 deletions

9
FAQ.md
View File

@ -133,9 +133,9 @@ Try with another USB cable or plug it into another USB port. See [#281] and
[#283]: https://github.com/Genymobile/scrcpy/issues/283 [#283]: https://github.com/Genymobile/scrcpy/issues/283
## HID/OTG issues on Windows ## OTG issues on Windows
On Windows, if `scrcpy --otg` (or `--hid-keyboard`/`--hid-mouse`) results in: On Windows, if `scrcpy --otg` (or `--keyboard=aoa`/`--mouse=aoa`) results in:
> ERROR: Could not find any USB device > ERROR: Could not find any USB device
@ -170,12 +170,13 @@ The default text injection method is [limited to ASCII characters][text-input].
A trick allows to also inject some [accented characters][accented-characters], A trick allows to also inject some [accented characters][accented-characters],
but that's all. See [#37]. but that's all. See [#37].
It is also possible to simulate a [physical keyboard][hid] (HID). To avoid the problem, [change the keyboard mode to simulate a physical
keyboard][hid].
[text-input]: https://github.com/Genymobile/scrcpy/issues?q=is%3Aopen+is%3Aissue+label%3Aunicode [text-input]: https://github.com/Genymobile/scrcpy/issues?q=is%3Aopen+is%3Aissue+label%3Aunicode
[accented-characters]: https://blog.rom1v.com/2018/03/introducing-scrcpy/#handle-accented-characters [accented-characters]: https://blog.rom1v.com/2018/03/introducing-scrcpy/#handle-accented-characters
[#37]: https://github.com/Genymobile/scrcpy/issues/37 [#37]: https://github.com/Genymobile/scrcpy/issues/37
[hid]: doc/hid-otg.md [hid]: doc/keyboard.md#physical-keyboard-simulation
## Client issues ## Client issues

View File

@ -32,10 +32,13 @@ Its features include:
- [configurable quality](doc/video.md) - [configurable quality](doc/video.md)
- [camera mirroring](doc/camera.md) (Android 12+) - [camera mirroring](doc/camera.md) (Android 12+)
- [mirroring as a webcam (V4L2)](doc/v4l2.md) (Linux-only) - [mirroring as a webcam (V4L2)](doc/v4l2.md) (Linux-only)
- [physical keyboard/mouse simulation (HID)](doc/hid-otg.md) - physical [keyboard][hid-keyboard] and [mouse][hid-mouse] simulation (HID)
- [OTG mode](doc/hid-otg.md#otg) - [OTG mode](doc/otg.md)
- and more… - and more…
[hid-keyboard]: doc/keyboard.md#physical-keyboard-simulation
[hid-mouse]: doc/mouse.md#physical-mouse-simulation
## Prerequisites ## Prerequisites
The Android device requires at least API 21 (Android 5.0). The Android device requires at least API 21 (Android 5.0).
@ -53,8 +56,7 @@ this option is set.
[control]: https://github.com/Genymobile/scrcpy/issues/70#issuecomment-373286323 [control]: https://github.com/Genymobile/scrcpy/issues/70#issuecomment-373286323
Note that USB debugging is not required to run scrcpy in [OTG Note that USB debugging is not required to run scrcpy in [OTG mode](doc/otg.md).
mode](doc/hid-otg.md#otg).
## Get the app ## Get the app
@ -73,11 +75,13 @@ documented in the following pages:
- [Video](doc/video.md) - [Video](doc/video.md)
- [Audio](doc/audio.md) - [Audio](doc/audio.md)
- [Control](doc/control.md) - [Control](doc/control.md)
- [Keyboard](doc/keyboard.md)
- [Mouse](doc/mouse.md)
- [Device](doc/device.md) - [Device](doc/device.md)
- [Window](doc/window.md) - [Window](doc/window.md)
- [Recording](doc/recording.md) - [Recording](doc/recording.md)
- [Tunnels](doc/tunnels.md) - [Tunnels](doc/tunnels.md)
- [HID/OTG](doc/hid-otg.md) - [OTG](doc/otg.md)
- [Camera](doc/camera.md) - [Camera](doc/camera.md)
- [Video4Linux](doc/v4l2.md) - [Video4Linux](doc/v4l2.md)
- [Shortcuts](doc/shortcuts.md) - [Shortcuts](doc/shortcuts.md)

View File

@ -10,36 +10,9 @@ scrcpy --no-control
scrcpy -n # short version scrcpy -n # short version
``` ```
## Keyboard and mouse
## Text injection preference Read [keyboard](keyboard.md) and [mouse](mouse.md).
Two kinds of [events][textevents] are generated when typing text:
- _key events_, signaling that a key is pressed or released;
- _text events_, signaling that a text has been entered.
By default, letters are injected using key events, so that the keyboard behaves
as expected in games (typically for WASD keys).
But this may [cause issues][prefertext]. If you encounter such a problem, you
can avoid it by:
```bash
scrcpy --prefer-text
```
(but this will break keyboard behavior in games)
On the contrary, you could force to always inject raw key events:
```bash
scrcpy --raw-key-events
```
These options have no effect on HID keyboard (all key events are sent as
scancodes in this mode).
[textevents]: https://blog.rom1v.com/2018/03/introducing-scrcpy/#handle-text-input
[prefertext]: https://github.com/Genymobile/scrcpy/issues/650#issuecomment-512945343
## Copy-paste ## Copy-paste
@ -85,6 +58,7 @@ way as <kbd>MOD</kbd>+<kbd>Shift</kbd>+<kbd>v</kbd>).
To disable automatic clipboard synchronization, use To disable automatic clipboard synchronization, use
`--no-clipboard-autosync`. `--no-clipboard-autosync`.
## Pinch-to-zoom, rotate and tilt simulation ## Pinch-to-zoom, rotate and tilt simulation
To simulate "pinch-to-zoom": <kbd>Ctrl</kbd>+_click-and-move_. To simulate "pinch-to-zoom": <kbd>Ctrl</kbd>+_click-and-move_.
@ -100,20 +74,7 @@ at a location inverted through the center of the screen. When pressing
<kbd>Ctrl</kbd> the x and y coordinates are inverted. Using <kbd>Shift</kbd> <kbd>Ctrl</kbd> the x and y coordinates are inverted. Using <kbd>Shift</kbd>
only inverts x. only inverts x.
This only works for the default mouse mode (`--mouse=sdk`).
## Key repeat
By default, holding a key down generates repeated key events. This can cause
performance problems in some games, where these events are useless anyway.
To avoid forwarding repeated key events:
```bash
scrcpy --no-key-repeat
```
This option has no effect on HID keyboard (key repeat is handled by Android
directly in this mode).
## Right-click and middle-click ## Right-click and middle-click
@ -147,7 +108,3 @@ The target directory can be changed on start:
```bash ```bash
scrcpy --push-target=/sdcard/Movies/ scrcpy --push-target=/sdcard/Movies/
``` ```
## Physical keyboard and mouse simulation
See the dedicated [HID/OTG](hid-otg.md) page.

View File

@ -234,7 +234,7 @@ The video and audio streams are decoded by [FFmpeg].
The client parses the command line arguments, then [runs one of two code The client parses the command line arguments, then [runs one of two code
paths][run]: paths][run]:
- scrcpy in "normal" mode ([`scrcpy.c`]) - scrcpy in "normal" mode ([`scrcpy.c`])
- scrcpy in [OTG mode](hid-otg.md) ([`scrcpy_otg.c`]) - scrcpy in [OTG mode](otg.md) ([`scrcpy_otg.c`])
[run]: https://github.com/Genymobile/scrcpy/blob/a3cdf1a6b86ea22786e1f7d09b9c202feabc6949/app/src/main.c#L81-L82 [run]: https://github.com/Genymobile/scrcpy/blob/a3cdf1a6b86ea22786e1f7d09b9c202feabc6949/app/src/main.c#L81-L82
[`scrcpy.c`]: https://github.com/Genymobile/scrcpy/blob/a3cdf1a6b86ea22786e1f7d09b9c202feabc6949/app/src/scrcpy.c#L292-L293 [`scrcpy.c`]: https://github.com/Genymobile/scrcpy/blob/a3cdf1a6b86ea22786e1f7d09b9c202feabc6949/app/src/scrcpy.c#L292-L293

View File

@ -1,112 +0,0 @@
# HID/OTG
By default, _scrcpy_ injects input events at the Android API level. As an
alternative, when connected over USB, it is possible to send HID events, so that
scrcpy behaves as if it was a physical keyboard and/or mouse connected to the
Android device.
A special [OTG](#otg) mode allows to control the device without mirroring (and
without USB debugging).
## Physical keyboard simulation
By default, _scrcpy_ uses Android key or text injection. It works everywhere,
but is limited to ASCII.
Instead, it can simulate a physical USB keyboard on Android to provide a better
input experience (using [USB HID over AOAv2][hid-aoav2]): the virtual keyboard
is disabled and it works for all characters and IME.
[hid-aoav2]: https://source.android.com/devices/accessories/aoa2#hid-support
However, it only works if the device is connected via USB.
Note: On Windows, it may only work in [OTG mode](#otg), not while mirroring (it
is not possible to open a USB device if it is already open by another process
like the _adb daemon_).
To enable this mode:
```bash
scrcpy --hid-keyboard
scrcpy -K # short version
```
If it fails for some reason (for example because the device is not connected via
USB), it automatically fallbacks to the default mode (with a log in the
console). This allows using the same command line options when connected over
USB and TCP/IP.
In this mode, raw key events (scancodes) are sent to the device, independently
of the host key mapping. Therefore, if your keyboard layout does not match, it
must be configured on the Android device, in Settings → System → Languages and
input → [Physical keyboard].
This settings page can be started directly:
```bash
adb shell am start -a android.settings.HARD_KEYBOARD_SETTINGS
```
However, the option is only available when the HID keyboard is enabled (or when
a physical keyboard is connected).
[Physical keyboard]: https://github.com/Genymobile/scrcpy/pull/2632#issuecomment-923756915
## Physical mouse simulation
By default, _scrcpy_ uses Android mouse events injection with absolute
coordinates. By simulating a physical mouse, a mouse pointer appears on the
Android device, and relative mouse motion, clicks and scrolls are injected.
To enable this mode:
```bash
scrcpy --hid-mouse
scrcpy -M # short version
```
When this mode is enabled, the computer mouse is "captured" (the mouse pointer
disappears from the computer and appears on the Android device instead).
Special capture keys, either <kbd>Alt</kbd> or <kbd>Super</kbd>, toggle
(disable or enable) the mouse capture. Use one of them to give the control of
the mouse back to the computer.
## OTG
It is possible to run _scrcpy_ with only physical keyboard and mouse simulation
(HID), as if the computer keyboard and mouse were plugged directly to the device
via an OTG cable.
In this mode, `adb` (USB debugging) is not necessary, and mirroring is disabled.
This is similar to `--hid-keyboard --hid-mouse`, but without mirroring.
To enable OTG mode:
```bash
scrcpy --otg
# Pass the serial if several USB devices are available
scrcpy --otg -s 0123456789abcdef
```
It is possible to enable only HID keyboard or HID mouse:
```bash
scrcpy --otg --hid-keyboard # keyboard only
scrcpy --otg --hid-mouse # mouse only
scrcpy --otg --hid-keyboard --hid-mouse # keyboard and mouse
# for convenience, enable both by default
scrcpy --otg # keyboard and mouse
```
Like `--hid-keyboard` and `--hid-mouse`, it only works if the device is
connected over USB.
## HID/OTG issues on Windows
See [FAQ](/FAQ.md#hidotg-issues-on-windows).

136
doc/keyboard.md Normal file
View File

@ -0,0 +1,136 @@
# Keyboard
Several keyboard input modes are available:
- `--keyboard=sdk` (default)
- `--keyboard=uhid` (or `-K`): simulates a physical HID keyboard using the UHID
kernel module on the device
- `--keyboard=aoa`: simulates a physical HID keyboard using the AOAv2 protocol
- `--keyboard=disabled`
By default, `sdk` is used, but if you use scrcpy regularly, it is recommended to
use [`uhid`](#uhid) and configure the keyboard layout once and for all.
## SDK keyboard
In this mode (`--keyboard=sdk`, or if the parameter is omitted), keyboard input
events are injected at the Android API level. It works everywhere, but it is
limited to ASCII and some other characters.
Note that on some devices, an additional option must be enabled in developer
options for this keyboard mode to work. See
[prerequisites](/README.md#prerequisites).
Additional parameters (specific to `--keyboard=sdk`) described below allow to
customize the behavior.
### Text injection preference
Two kinds of [events][textevents] are generated when typing text:
- _key events_, signaling that a key is pressed or released;
- _text events_, signaling that a text has been entered.
By default, numbers and "special characters" are inserted using text events, but
letters are injected using key events, so that the keyboard behaves as expected
in games (typically for WASD keys).
But this may [cause issues][prefertext]. If you encounter such a problem, you
can inject letters as text (or just switch to [UHID](#uhid)):
```bash
scrcpy --prefer-text
```
(but this will break keyboard behavior in games)
On the contrary, you could force to always inject raw key events:
```bash
scrcpy --raw-key-events
```
[textevents]: https://blog.rom1v.com/2018/03/introducing-scrcpy/#handle-text-input
[prefertext]: https://github.com/Genymobile/scrcpy/issues/650#issuecomment-512945343
### Key repeat
By default, holding a key down generates repeated key events. Ths can cause
performance problems in some games, where these events are useless anyway.
To avoid forwarding repeated key events:
```bash
scrcpy --no-key-repeat
```
## Physical keyboard simulation
Two modes allow to simulate a physical HID keyboard on the device.
To work properly, it is necessary to configure (once and for all) the keyboard
layout on the device to match that of the computer.
The configuration page can be opened in one of the following ways:
- from the scrcpy window (when `uhid` or `aoa` is used), by pressing
<kbd>MOD</kbd>+<kbd>k</kbd> (see [shortcuts](shortcuts.md))
- from the device, in Settings → System → Languages and input → Physical
devices
- from a terminal on the computer, by executing `adb shell am start -a
android.settings.HARD_KEYBOARD_SETTINGS`
From this configuration page, it is also possible to enable or disable on-screen
keyboard.
### UHID
This mode simulates a physical HID keyboard using the [UHID] kernel module on the
device.
[UHID]: https://kernel.org/doc/Documentation/hid/uhid.txt
To enable UHID keyboard, use:
```bash
scrcpy --keyboard=uhid
scrcpy -K # short version
```
Once the keyboard layout is configured (see above), it is the best mode for
using the keyboard while mirroring:
- it works for all characters and IME (contrary to `--keyboard=sdk`)
- the on-screen keyboard can be disabled (contrary to `--keyboard=sdk`)
- it works over TCP/IP (wirelessly) (contrary to `--keyboard=aoa`)
- there are no issues on Windows (contrary to `--keyboard=aoa`)
One drawback is that it may not work on old Android versions due to permission
errors.
### AOA
This mode simulates a physical HID keyboard using the [AOAv2] protocol.
[AOAv2]: https://source.android.com/devices/accessories/aoa2#hid-support
To enable AOA keyboard, use:
```bash
scrcpy --keyboard=aoa
```
Contrary to the other modes, it works at the USB level directly (so it only
works over USB).
It does not use the scrcpy server, and does not require `adb` (USB debugging).
Therefore, it is possible to control the device (but not mirror) even with USB
debugging disabled (see [OTG](otg.md)).
Note: On Windows, it may only work in [OTG mode](otg.md), not while mirroring
(it is not possible to open a USB device if it is already open by another
process like the _adb daemon_).

70
doc/mouse.md Normal file
View File

@ -0,0 +1,70 @@
# Mouse
Several mouse input modes are available:
- `--mouse=sdk` (default)
- `--mouse=uhid` (or `-M`): simulates a physical HID mouse using the UHID
kernel module on the device
- `--mouse=aoa`: simulates a physical HID mouse using the AOAv2 protocol
- `--mouse=disabled`
## SDK mouse
In this mode (`--mouse=sdk`, or if the parameter is omitted), mouse input events
are injected at the Android API level with absolute coordinates.
Note that on some devices, an additional option must be enabled in developer
options for this mouse mode to work. See
[prerequisites](/README.md#prerequisites).
## Physical mouse simulation
Two modes allow to simulate a physical HID mouse on the device.
In these modes, the computer mouse is "captured": the mouse pointer disappears
from the computer and appears on the Android device instead.
Special capture keys, either <kbd>Alt</kbd> or <kbd>Super</kbd>, toggle
(disable or enable) the mouse capture. Use one of them to give the control of
the mouse back to the computer.
### UHID
This mode simulates a physical HID mouse using the [UHID] kernel module on the
device.
[UHID]: https://kernel.org/doc/Documentation/hid/uhid.txt
To enable UHID mouse, use:
```bash
scrcpy --mouse=uhid
scrcpy -M # short version
```
### AOA
This mode simulates a physical HID mouse using the [AOAv2] protocol.
[AOAv2]: https://source.android.com/devices/accessories/aoa2#hid-support
To enable AOA mouse, use:
```bash
scrcpy --mouse=aoa
```
Contrary to the other modes, it works at the USB level directly (so it only
works over USB).
It does not use the scrcpy server, and does not require `adb` (USB debugging).
Therefore, it is possible to control the device (but not mirror) even with USB
debugging disabled (see [OTG](otg.md)).
Note: On Windows, it may only work in [OTG mode](otg.md), not while mirroring
(it is not possible to open a USB device if it is already open by another
process like the _adb daemon_).

37
doc/otg.md Normal file
View File

@ -0,0 +1,37 @@
# OTG
By default, _scrcpy_ injects input events at the Android API level. As an
alternative, when connected over USB, it is possible to send HID events, so that
scrcpy behaves as if it was a physical keyboard and/or mouse connected to the
Android device.
A special mode allows to control the device without mirroring, using AOA
[keyboard](keyboard.md#aoa) and [mouse](mouse.md#aoa). Therefore, it is possible
to run _scrcpy_ with only physical keyboard and mouse simulation (HID), as if
the computer keyboard and mouse were plugged directly to the device via an OTG
cable.
In this mode, `adb` (USB debugging) is not necessary, and mirroring is disabled.
This is similar to `--keyboard=aoa --mouse=aoa`, but without mirroring.
To enable OTG mode:
```bash
scrcpy --otg
# Pass the serial if several USB devices are available
scrcpy --otg -s 0123456789abcdef
```
It is possible to disable HID keyboard or HID mouse:
```bash
scrcpy --otg --keyboard=disabled
scrcpy --otg --mouse=disabled
```
It only works if the device is connected over USB.
## OTG issues on Windows
See [FAQ](/FAQ.md#otg-issues-on-windows).