Support Keyboard / Switch instead of Touch

[ImpulseAdventure]

GUIslice is currently most useful for displays that have touchscreens. However, it should also be possible to enable the GUI elements to be selected using either a keyboard (for Raspberry Pi) or buttons / switches (on Arduino, M5Stack and similar devices).

I was thinking of adding modes for these (such as DRV_TOUCH_KEY and DRV_TOUCH_PIN) and they would probably operate along these ways:

• A "next" key/button moves focus to the next GUI element, skipping over any elements that can't accept focus
• A "prev" key/button moves focus to the previous GUI element, skipping over any elements that can't accept focus
• A "select" key/button selects the currently focused GUI element
• The focused GUI element is shown by using the "glowing" color attribute
• The display begins with no element in focus
• One the next/prev advances past the last GUI element, it goes back to "no elements in focus" and then wraps back to the start
• Selecting an element will make the glowing state return to the normal fill state, and carry out the action
• Later, I might explore ways to handle sliders as well
• Elements would have a config bit that indicates if it can accept focus (eg. text elements can't, but buttons, checkboxes can, etc.)
• Focus order would be dictated solely by the element creation order (at least for now)
• Keyboard / pin binding would be maintained in a separate mapping structure

With the above, it may be possible to expand the usability of GUIslice to many other displays. In fact, I could enable the M5Stack driver to connect directly to this capability as the device has integrated buttons. E-ink displays could also benefit as they usually don't have touch screens.

Any feedback or suggestions on this idea would be greatly appreciated!

[ImpulseAdventure]

Current status: I now have a working prototype that enables keyboard navigation between the selectable UI elements. This currently works with a keyboard in SDL1 mode, but it should be easily extended to accept an API call from GPIO button inputs in Arduino mode.

I need further thought on how to best support keyboard / physical button control over dynamic elements like sliders. I’m open to suggestions!

[ImpulseAdventure]

Keyboard control (in SDL1 mode) is now working for the basic UI elements (eg. buttons, checkboxes, radio buttons) as well as the slider element.

The user code can register key mappings to GUI actions. The GUI actions currently include the following:

• Focus Next (change current element focus)
• Focus Previous (change current element focus)
• Select (select focused element0
• Set Value Relative (adjusts the value of a control such as slider by a relative amount)
• Set Value Absolute (adjusts the value of a control such as slider to an absolute value)

For example, the following code is all it takes to add keyboard control to the GUI:

gslc_InitInputMap(&m_gui,m_asInputMap,MAX_INPUT_MAP);
gslc_InputMapKeyAdd(&m_gui,SDLK_UP,GSLC_ACTION_FOCUS_PREV,0);
gslc_InputMapKeyAdd(&m_gui,SDLK_DOWN,GSLC_ACTION_FOCUS_NEXT,0);
gslc_InputMapKeyAdd(&m_gui,SDLK_SPACE,GSLC_ACTION_SELECT,0);

On devices that are very constrained in number of physical buttons (eg. 3 or fewer physical buttons / keys), navigation is still possible, but adjusting controls such as sliders requires some further thought. Perhaps the select button (when engaged on a slider control) would swap between "focus navigate" mode and "set value" mode. Once the value has been configured, the select button is used again to return to navigate mode.

[ImpulseAdventure]

Preliminary support for GPIO button controls over the GUI has now been implemented (for all devices), in addition to the user-defined keyboard controls (for SDL mode). Once I have incorporated pin input debouncing and completed some further testing, I'll be able to integrate the feature into the repo soon.

All in all, these features should enable GUIslice to be a lot more useful when driving non-touch displays such as OLEDs and e-Ink.

[ImpulseAdventure]

Support for GPIO & keyboard control has now been integrated into the master branch (in abff04b).

An example (ex21) has been created to demonstrate its usage:

• /arduino/ex21_ard_input_pin.ino: (GPIO pin control)
• /linux/ex22_lnx_input_key.c: (Keyboard control in SDL1.2 mode)

The associated APIs should be regarded as tentative at the moment and subject to change, but should be stable enough to test out.

Snippet of example code to use GPIO pin control over GUIslice (in Arduino mode):

#define PIN_PREV  38
#define PIN_SEL   40
#define PIN_NEXT  42
EasyButton btn_prev(PIN_PREV);
EasyButton btn_sel( PIN_SEL);
EasyButton btn_next(PIN_NEXT);
...
  // Set the pin poll callback function
  gslc_SetPinPollFunc(&m_gui, CbPinPoll);

  // Create the GUI input mapping (pin event to GUI action)
  gslc_InitInputMap(&m_gui,m_asInputMap,MAX_INPUT_MAP);
  gslc_InputMapAdd(&m_gui, GSLC_INPUT_PIN_ASSERT, PIN_PREV, GSLC_ACTION_FOCUS_PREV, 0);
  gslc_InputMapAdd(&m_gui, GSLC_INPUT_PIN_ASSERT, PIN_SEL,  GSLC_ACTION_SELECT, 0);
  gslc_InputMapAdd(&m_gui, GSLC_INPUT_PIN_ASSERT, PIN_NEXT, GSLC_ACTION_FOCUS_NEXT, 0);

Notes on the examples:

• In all cases, GSLC_FEATURE_INPUT has to be set to 1 in the GUIslice_config_*.h file to enable the new feature.
• The Arduino GPIO control example leverages the EasyButton debouncing code (included in the ex21_ard_input_pin directory), but any other library that provides polling access can be used. I might also offer an alternate method later that utilizes callbacks from a debounce/button library.
• The LINUX/Raspberry Pi keyboard control utilizes SDL key handling, so GSLC_TOUCH_SDL needs to be enabled in the GUIslice_config_linux.h. Note that SDL1.2 has some issues with PiTFT displays (unrelated to GUIslice), so tslib is generally the recommended mode for touch displays.
• The LINUX/Raspberry Pi GPIO control will be added to the ex21_lnx_input_pin example later.

[ImpulseAdventure]

Support has now been added (in local development main repo) for M5stack's integrated buttons.
The 3 physical buttons can navigate the GUI and also modify the checkboxes, radio buttons and sliders. Button A changes focus to previous element, Button C to next element and Button B selects an element (eg. button select, toggle a checkbox, etc.)

Decrement / increment on slider controls is currently mapped to long-presses on button A & C, but I am exploring some other ideas that might involve introducing a new element focus state ("edit" state?) that then uses A & C for decrement / increment. Button B long-press could potentially be used to toggle between focus and edit states.

For M5stack, since the button handlers are already built-in to its library, adding user control over the GUI is very easy with only the following lines added to one's sketch:

  gslc_InitInputMap(&m_gui, m_asInputMap, MAX_INPUT_MAP);
  gslc_InputMapAdd(&m_gui, GSLC_INPUT_PIN_ASSERT, GSLC_PIN_BTN_A, GSLC_ACTION_FOCUS_PREV, 0);
  gslc_InputMapAdd(&m_gui, GSLC_INPUT_PIN_ASSERT, GSLC_PIN_BTN_B, GSLC_ACTION_SELECT, 0);
  gslc_InputMapAdd(&m_gui, GSLC_INPUT_PIN_ASSERT, GSLC_PIN_BTN_C, GSLC_ACTION_FOCUS_NEXT, 0);

The remaining issue to resolve is how to indicate to the user the difference between the focus state and edit state (I am currently using the "glowing" element state to indicate the GUI focus).

@Sakabin or @0x1abin -- if either of you are interested in testing out this feature or have ideas on a suitable convention for how you'd prefer to see the M5stack buttons navigate a GUI, please let me know. (I also have a simplified GUIslice config file specific to M5stack here: m5-m5-default-btn3.h )

thanks!

[ImpulseAdventure]

Support for button control without any existing touch drivers (ie. DRV_TOUCH_NONE) is now possible.

• In the config file, specify DRV_TOUCH_INPUT which indicates that no external touch drivers are installed, but that GPIO/button handling support is requested.
• An example config ard-adagfx-ili9341-input.h has been provided to demonstrate this.