Replace screenrecord execution by manual screen encoding using the
MediaCodec API.
The "screenrecord" solution had several drawbacks:
- screenrecord output is buffered, so tiny frames may not be accessible
immediately;
- it did not output a frame until the surface changed, leading to a
black screen on start;
- it is limited to 3 minutes recording, so it needed to be restarted;
- screenrecord added black borders in the video when the requested
dimensions did not preserve aspect-ratio exactly (sometimes
unavoidable since video dimensions must be multiple of 8);
- rotation handling was hacky (killing the process and starting a new
one).
Handling the encoding manually allows to solve all these problems.
Accept a parameter to limit the video size.
For instance, with "-m 960", the great side of the video will be scaled
down to 960 (if necessary), while the other side will be scaled down so
that the aspect ratio is preserved. Both dimensions must be a multiple
of 8, so black bands might be added, and the mouse positions must be
computed accordingly.
The video screen size on the client may differ from the real device
screen size (e.g. the video stream may be scaled down). As a
consequence, mouse events must be scaled to match the real device
coordinates.
For this purpose, make the client send the video screen size along with
the absolute pointer location, and the server scale the location to
match the real device size before injecting mouse events.
Currently, we only use screen information (width, height, rotation)
once at initialization, to send the device size to the client.
To be able to scale mouse events, make it accessible in memory. For this
purpose, replace the "static" DeviceUtil to a singleton Device, and
update it on every screen rotation.
Ctrl, Alt, Shift and Meta should not be transmitted to the Android
device: they may generate unwanted events. For instance, resizing the
window using Alt+click will generate an Alt event which may open a menu
on the device.
All keycodes that generate a text input must also be excluded, to avoid
the text input to be written twice.
To control the device from the computer:
- retrieve mouse and keyboard SDL events;
- convert them to Android events;
- serialize them;
- send them on the same socket used by the video stream (but in the
opposite direction);
- deserialize the events on the Android side;
- inject them using the InputManager.
exit() should not be called from within a child process, since it would
call functions registered with atexit(), and flush stdio streams. Use
_exit() instead.
Move the DeviceUtil internal static classes to public classes, in a
separate package (".wrappers").
This paves the way to implement InputManager properly.
Romain Vimont