Video-audio speller

Transforms a Raspberry Pi Zero (RPi0) into a keyboard operable by a user with motor disabilities.

The RPi0 is connected to a host computer through its USB communication port, as well as to a video/audio device through its mini-HDMI port. A simple push-button (normally open monostable switch) is also connected between the ground and pin 21 of the RPi0 GPIO. The host computer provides the power through the USB connection.

On the one hand, from the host computer point of view, the RPi0 appears as a standard USB keyboard. On the other hand, the RPi0 displays on the device connected to its HDMI output a matrix of symbols that are continuously scanned until one of them is selected, which occurs when the push-button is activated.

Symbols are logically organized as a tree in which the user navigates. When a specific tree node is active, its children are scanned sequentially until the user selects one of them. If the selected child is another tree node, it becomes the active node. If the selected child is a leaf, i.e. a terminal tree node, one or several scan-codes are sent to the host computer, simulating specific keystrokes. The root node of the tree is then reactivated to start a new selection process.

On the attached HDMI device, several rows of symbols are displayed. The first row displays the symbols associated with all the children of the root node. When the root node is scanned, this first displayed row is active and each child symbol is highlighted (displayed in yellow) every two seconds. When a symbol is highlighted, an optional associated sound is also played on the audio output, allowing users with visual impairments to use the system. If the highlighted symbol corresponds to a tree node with children, all the symbols associated with them are displayed on the next row.

When a symbol of the active row is selected, if its tree node has no child, associated keystrokes are sent to the host computer as described before and the process starts over. When the tree node associated with the selection has children, the symbol is now displayed in red and the next row of symbols becomes active. This allows for using several depth levels for each tree branch.

A resting state (pause) stops the scanning process when the user does not select a symbol during two successive scanning cycles of the active node. During the resting state, the display only shows the symbols associated with the root node. To restart the scanning process, the user must activate the push-button once. At boot, the system is in the resting state, therefore a first activation of the push-button is required.

Installation on a Raspberry Pi Zero

The installation steps described hereafter have been checked on a Raspberry Pi Zero W V1.1, after a fresh install of a 32bit version of Raspbian (bullseye v11, kernel v5.15, release date: April 4th 2022). For Raspbian installation instructions, see https://www.raspberrypi.com/software/.

Required additional modules

• wxPython

This module provides the GUI used by the application. wxPython is available on all standard operating systems, which allows an easy testing and configuration of the software on any computer. To install wxPython execute:

sudo pip install wxPython

• pypubsub

This python module allows communication between processes through a publish-subscribe mechanism, see https://pypubsub.readthedocs.io/. To install pypubsub execute:

sudo pip install pypubsub

HID keyboard configuration

The following configuration steps are strongly inspired by the tutorial available at http://www.isticktoit.net/?p=1383. The aim is to make the RPi0 appear as a USB keyboard when connected to a host computer.

First, modules must be added to the kernel to enable a device tree overlay:

echo "dtoverlay=dwc2" | sudo tee -a /boot/config.txt
echo "dwc2" | sudo tee -a /etc/modules
echo "libcomposite" | sudo tee -a /etc/modules

Since gadget devices definitions are volatile, the HID keyboard must be created after each boot by a configuration script. To create this script execute:

sudo touch /usr/bin/cristal-speller
sudo chmod +x /usr/bin/cristal-speller

Then edit this script using:

sudo nano /usr/bin/cristal-speller

and copy/paste the following device definition parameters:

#!/bin/bash
cd /sys/kernel/config/usb_gadget/
# Main
mkdir -p cristal
cd cristal
echo 0x1d6b > idVendor # Linux Foundation
echo 0x0104 > idProduct # Multifunction Composite Gadget
echo 0x0100 > bcdDevice # v1.0.0
echo 0x0200 > bcdUSB # USB2
# Strings
mkdir -p strings/0x409
echo "2022-04-29-0001" > strings/0x409/serialnumber
echo "CRIStAL" > strings/0x409/manufacturer
echo "CRIStAL - Univ. Lille - USB Keyboard" > strings/0x409/product
# Configurations
mkdir -p configs/config.1/strings/0x409
echo "Config 1: USB Keyboard" > configs/config.1/strings/0x409/configuration
echo 170 > configs/config.1/MaxPower
# Keyboard functions
mkdir -p functions/hid.usb0
echo 1 > functions/hid.usb0/protocol
echo 1 > functions/hid.usb0/subclass
echo 8 > functions/hid.usb0/report_length
echo -ne \\x05\\x01\\x09\\x06\\xa1\\x01\\x05\\x07\\x19\\xe0\\x29\\xe7\\x15\\x00\\x25\\x01\\x75\\x01\\x95\\x08\\x81\\x02\\x95\\x01\\x75\\x08\\x81\\x03\\x95\\x05\\x75\\x01\\x05\\x08\\x19\\x01\\x29\\x05\\x91\\x02\\x95\\x01\\x75\\x03\\x91\\x03\\x95\\x06\\x75\\x08\\x15\\x00\\x25\\x65\\x05\\x07\\x19\\x00\\x29\\x65\\x81\\x00\\xc0 > functions/hid.usb0/report_desc
# Associations
ln -s functions/hid.usb0 configs/config.1/
# Activate device
ls /sys/class/udc > UDC
# Change device access rights
chmod a+rw /dev/hidg0

Finally, save (ctrl+O) and close (ctrl+X) the editor.

The device definition script is called at each system startup by the /etc/rc.local script. This call is inserted in /etc/rc.local just before the exit instruction by executing:

sudo sed -i '/^exit.*/i /usr/bin/cristal-speller # libcomposite configuration' /etc/rc.local

Application installation

First, get a copy of the application from this gitlab page and uncompress all the files into a dedicated folder, keeping the directory structure, by executing:

cd ~ && curl -L https://gitlab.cristal.univ-lille.fr/fcabesta/video-audio-speller/-/archive/master/video-audio-speller-master.tar.gz | tar xz

Then, in order to start the application after GUI boot, create a shortcut by executing:

mkdir ~/.config/autostart
nano ~/.config/autostart/cristal-speller.desktop

and copy/paste the following shortcut parameters:

[Desktop Entry]
Type=Application
Name=CRIStAL Speller
Exec=/bin/bash -c 'cd ~/video-audio-speller-master && python speller.py'

Finally, save (ctrl+O) and close (ctrl+X) the editor.

Final adjustments

Since there will be no user input on the RPi0 through a keyboard or a mouse, we need to block the automatic screen blanking of the GUI, by executing:

sudo sed -i '/^# xserver-command.*/a xserver-command=X -s 0 -dpms' /etc/lightdm/lightdm.conf

You are done! After reboot, your Raspberry Pi Zero should work as indicated at the beginning of this page!