Take screenshots while playing the game and record finger actions. For example, data is collected every 100 milliseconds, and each record includes a screenshot and the status of the finger operation. We hope that the trained AI can predict the state of finger operation in the next moment, and then reflect the predicted operation into the game.
cell phone
Android phones, do not consider iOS.
Take a screenshot of your phone
You can record the game process as a video file by recording the screen on your mobile phone, and then get a screenshot from the video file. General mobile phones come with a screen recording function. It is also possible to record screens from a computer, such as scrcpy(). Generally, we can't record sound, but we only use screenshots, so we don't have to record sound.
Record finger actions
The tool gettevent that comes with Android can get a variety of input events, including touch events on the screen. On a computer, you can call it with the adb command (if you are not familiar with the adb command, please search for it yourself).
$ adb shell getevent -lp
// 输出差不多这样子
add device 6: /dev/input/event1
name: "hisi_gpio_key"
events:
KEY (0001): KEY_VOLUMEDOWN KEY_VOLUMEUP
input props:
<none>
add device 7: /dev/input/event6
name: "huawei,ts_kit"
events:
KEY (0001): KEY_F1 BTN_TOOL_FINGER BTN_TOUCH
ABS (0003): ABS_MT_TOUCH_MAJOR : value 0, min 0, max 255, fuzz 0, flat 0, resolution 0
ABS_MT_TOUCH_MINOR : value 0, min 0, max 255, fuzz 0, flat 0, resolution 0
ABS_MT_WIDTH_MAJOR : value 0, min 0, max 100, fuzz 0, flat 0, resolution 0
ABS_MT_WIDTH_MINOR : value 0, min 0, max 100, fuzz 0, flat 0, resolution 0
ABS_MT_ORIENTATION : value 0, min 0, max 255, fuzz 0, flat 0, resolution 0
ABS_MT_POSITION_X : value 0, min 0, max 1080, fuzz 0, flat 0, resolution 0
ABS_MT_POSITION_Y : value 0, min 0, max 1920, fuzz 0, flat 0, resolution 0
ABS_MT_TRACKING_ID : value 0, min 0, max 15, fuzz 0, flat 0, resolution 0
ABS_MT_PRESSURE : value 0, min 0, max 1080, fuzz 0, flat 0, resolution 0 "adb shell getevent -lp" lists all input devices, and the device ID is similar to "/dev/input/event...". Find the device that starts with ABS_MT in the events, and it's the touchscreen. Record the device ID, and then we'll get the event for the touchscreen.
$ adb shell getevent -l /dev/input/event6
// 输出差不多这样子
// type, code, value
EV_SYN SYN_MT_REPORT 00000000
EV_SYN SYN_REPORT 00000000
EV_ABS ABS_MT_PRESSURE 00000076
EV_ABS ABS_MT_POSITION_X 00000249
EV_ABS ABS_MT_POSITION_Y 00000508
EV_ABS ABS_MT_TRACKING_ID 00000000
EV_SYN SYN_MT_REPORT 00000000
EV_KEY BTN_TOUCH DOWN
EV_SYN SYN_REPORT 00000000
EV_ABS ABS_MT_PRESSURE 00000063
EV_ABS ABS_MT_POSITION_X 00000249
EV_ABS ABS_MT_POSITION_Y 00000508
EV_ABS ABS_MT_TRACKING_ID 00000000
EV_SYN SYN_MT_REPORT 00000000
EV_SYN SYN_REPORT 00000000 You can search for multi-touch protocols to understand the meaning of the above output, such as this one
More code for recording touch screen actions can be found here
Time alignment
Since we record the screen and touch screen operations separately, when processing the data later, we need to synchronize the screenshot with the operation at the same moment, so when recording the original data, we try to carry a timestamp as much as possible.