Demostrate Slint integration with Zephyr OS #5411

0x6e · 2024-06-14T16:02:32Z

Using the existing printerdemo_mcu example, demonstrate how to implement slint::platform::Platform and slint::platform::WindowAdapter for Zephyr OS. The demo has been tested with the Zephyr native simulator, and the NXP MIMXRT1170-EVKB with a RK055HDMIPI4MA0 MIPI display.

Know issues:

unlike the Espressif integration, we don't provide the platform integration as part of the Slint C++ API. In part, this is due to the way Zephyr OS handles device hardware. Zephyr uses the Device Tree to describe the hardware to the device driver model. In order to register an input event call back we need a pointer to a device obtained from a device tree node, and we also need to know how the driver behaves in order to write our callback function. The existing implementation is generic enough to cover the simulator and display shield drivers. A more general solution could be investigated in the future;
double buffering is not supported as neither the simulator or the hardware used for testing reported it as supported;
in the simulator, we convert dirty regions to big-endian. However, where regions overlap, the intersections get converted more than once resulting in incorrect colors on the display. This is a general issue caused by overlapping dirty regions, and should be tackled separately as it also affects the Espressif demo;
if there are active animations, we need to make sure to sleep for a fixed period of time, otherwise the event loop runs forever, never re-rendering and never getting to a state where there are no active animations. I haven't figured out whether this is an issue with the async code in the platform integration, or something I missed regarding threading with Zephyr in particular (i.e. I haven't tried other schedulers yet);
The example doesn't use the full screen on the hardware. We should probably use the non-mcu version of printerdemo. For this to work nicely we need to rotate the screen to landscape mode.

For now I would like to get the existing code reviewed, and look at fixing the issues in follow up PRs.

Add a variant of the printerdemo_mcu that works with Zephyr. For now we only target the native_sim, and only have rendering set up. Future commits will add touch support, and investigate synchronization issues.

Don't forget to source the Zephyr virtual env.

Use the correct offset into the buffer. In combination with zephyrproject-rtos/zephyr#72007 partial rendering now works as expected.

Turn on Zephyr asserts to help root out issues with the Zephyr platform code.

This stops the display from flickering in the native_sim.

Define a simple callback to handle touch events from Zephyr. We expect an input_sdl_touch device to exist in the device tree. This is a fairly simple implementation as a proof of concept. The default behaviour of Zephyr (as configured) is to handle inputs in a separate thread. Each `input_event` describes a specific change, with the `sync` member specifying when the event has reached a stable state. We use slint::invoke_from_event_loop to dispatch the slint event in the main thread.

Make sure we sleep, even if there are animations. Previously we would spin the event loop forever once an animation started. Ideally we need some slint API to get the duration until the next animation tick.

Zephyr does not support std::mutex [1]. [1] https://docs.zephyrproject.org/latest/develop/languages/cpp/index.html#language-features

Setup the printerdemo_mcu example to build and run on the NXP MIMXRT1170-EVKB. There is an issue with the supported pixel formats, which will be resolved in a future commit

Use the correct format for the duration representation.

Zephyr reports that the RK055HDMIPI4MA0 MIPI display is using PIXEL_FORMAT_BGR_565. The buffer we provide to Slint uses `slint::platform::Rgb565Pixel`. Somewhere, the pixel data ends up in the correct format to display on the test hardware.

Slint is taking quite some time to update the render buffer: ``` // Initial render [00:00:01.727,000] <dbg> zephyrSlint: maybe_redraw: Rendered x: 0 y: 0 w: 720 h: 1280 [00:00:01.727,000] <dbg> zephyrSlint: maybe_redraw: - total: 1363 ms, slint: 1347 ms, write: 16 ms [00:00:01.727,000] <dbg> zephyrSlint: run_event_loop: Sleeping for 0ms [00:00:01.727,000] <dbg> zephyrSlint: run_event_loop: Loop // Partial update [00:00:02.255,000] <dbg> zephyrSlint: maybe_redraw: Rendered x: 146 y: 18 w: 156 h: 204 [00:00:02.255,000] <dbg> zephyrSlint: maybe_redraw: - total: 525 ms, slint: 504 ms, write: 21 ms [00:00:02.255,000] <dbg> zephyrSlint: run_event_loop: Sleeping for 474ms ```

Render times are greatly improved in release builds.

Zephyr expects pixel data to be big endian [1]. The display driver expects RGB 565 pixel data [2], and appears to expect it to be little endian. By passing Slint's little endian, RGB 565 pixel data without converting to big endian as Zephyr expects, we get colors that work. [1] https://docs.zephyrproject.org/latest/hardware/peripherals/display/index.html#c.display_pixel_format [2] https://github.com/zephyrproject-rtos/zephyr/blob/c211cb347e0af0a4931e0e7af3d93577bcc7af8f/drivers/display/display_mcux_elcdif.c#L256

Track the Zephyr v3.6.0 release. Update the documentation to make use of the West manifest.

For the native sim and the NXP MIMXRT1170-EVKB: - checkout the code; - install the linux dependencies, including Zephyr dependencies; - setup rust with the correct target for the board; - setup the Zephyr project, using the printerdemo_mcu manifest; - export the Zephyr CMake package; - build for the board specified in the matrix.

The native simulator uses the SDL display driver. In Zephyr v3.6.0 this driver incorrectly mixes pitch and width in asserts, causing the application to assert when run. This revision fixes the assert. Ideally we would use a tagged release, but there is not one available yet.

The hardware model in Zephyr has changed. Use the way to specify a board.

The performance is much better, so for someone testing the demo based on the readme, this gives a better first experience.

The software renderer now provides a list of regions which have changed. Convert the regions to big-endian and render them individually to prevent the non-dirty regions from being converted away from big-endian. Unfortunately, this still means we convert any intersection between region more than once but this will be tackled as a follow-up commit, as in principle this affects more than just the Zephyr demo.

If there are no active animations or timer updates, allow the event loop to run forever, rather than hardcoding five seconds.

The Zephyr VERSION file is excluded as this simply specifies the version of the Zephyr applicaiton, and the documentation doesn't say how comments should be specified or if they are supported.

Use the latest available SDK.

tronical

Overall this looks good to me. It's very well contained. @ogoffart any thoughts? :)

examples/printerdemo_mcu/zephyr/README.md

Include the current know issues in the documentation, so that they aren't hidden in a pull request.

Direct the reader to the same version that the CI uses.

0x6e force-pushed the feature/zephyr-printerdemo branch 4 times, most recently from a14fc4c to d707c08 Compare June 28, 2024 15:07

0x6e added 16 commits June 28, 2024 15:08

Render printerdemo in Zephyr RTOS

f527af9

Add a variant of the printerdemo_mcu that works with Zephyr. For now we only target the native_sim, and only have rendering set up. Future commits will add touch support, and investigate synchronization issues.

Update examples/printerdemo_mcu/zephyr/README.md

f098e70

Don't forget to source the Zephyr virtual env.

Fix ZephyrWindowAdapter rendering

afbaf35

Use the correct offset into the buffer. In combination with zephyrproject-rtos/zephyr#72007 partial rendering now works as expected.

Enable asserts in Zephyr

91e7dfd

Turn on Zephyr asserts to help root out issues with the Zephyr platform code.

Zephyr: Turn off display blanking once

2103eb9

This stops the display from flickering in the native_sim.

Zephyr: don't block main thread with animations

c8b1313

Make sure we sleep, even if there are animations. Previously we would spin the event loop forever once an animation started. Ideally we need some slint API to get the duration until the next animation tick.

Zephyr: don't use std::mutex

1ceff94

Zephyr does not support std::mutex [1]. [1] https://docs.zephyrproject.org/latest/develop/languages/cpp/index.html#language-features

Build and run on NXP MIMXRT1170-EVKB

f2417b8

Setup the printerdemo_mcu example to build and run on the NXP MIMXRT1170-EVKB. There is an issue with the supported pixel formats, which will be resolved in a future commit

Zephyr: fix format specifier warning

a3e8338

Use the correct format for the duration representation.

Zephyr: prefer a release build on device

a416bd7

Render times are greatly improved in release builds.

Add a West manifest

55d2023

Track the Zephyr v3.6.0 release. Update the documentation to make use of the West manifest.

0x6e force-pushed the feature/zephyr-printerdemo branch from d707c08 to bf9790c Compare June 28, 2024 15:08

0x6e changed the title ~~WIP: feature/zephyr printerdemo~~ Demostrate Slint integration with Zephyr OS Jun 28, 2024

0x6e added 5 commits June 30, 2024 07:54

Zephyr: fix use of deprecated board specification

0b46464

The hardware model in Zephyr has changed. Use the way to specify a board.

Zephyr: prefer a release build for hardware

1ba739b

The performance is much better, so for someone testing the demo based on the readme, this gives a better first experience.

Zephyr: allow the event loop to wait forever

6a911d1

If there are no active animations or timer updates, allow the event loop to run forever, rather than hardcoding five seconds.

0x6e force-pushed the feature/zephyr-printerdemo branch 2 times, most recently from ea87133 to 0ee51d6 Compare June 30, 2024 08:40

Zephyr: apply license headers

be49b9b

The Zephyr VERSION file is excluded as this simply specifies the version of the Zephyr applicaiton, and the documentation doesn't say how comments should be specified or if they are supported.

Zephyr: bump the SDK version used by the mcu_zephyr task

a887cf5

Use the latest available SDK.

0x6e force-pushed the feature/zephyr-printerdemo branch from 0ee51d6 to a887cf5 Compare June 30, 2024 08:56

0x6e marked this pull request as draft June 30, 2024 09:40

0x6e marked this pull request as ready for review June 30, 2024 09:40

tronical approved these changes Jul 2, 2024

View reviewed changes

examples/printerdemo_mcu/zephyr/README.md Show resolved Hide resolved

0x6e added 2 commits July 2, 2024 17:03

Zephyr: update the README.md with known issues

b3af673

Include the current know issues in the documentation, so that they aren't hidden in a pull request.

Zephyr: update Zephyr SDK version in README.md

59dcda4

Direct the reader to the same version that the CI uses.

tronical approved these changes Jul 8, 2024

View reviewed changes

tronical merged commit 3d5a876 into slint-ui:master Jul 8, 2024
38 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Demostrate Slint integration with Zephyr OS #5411

Demostrate Slint integration with Zephyr OS #5411

0x6e commented Jun 14, 2024 •

edited

Loading

tronical left a comment

Demostrate Slint integration with Zephyr OS #5411

Demostrate Slint integration with Zephyr OS #5411

Conversation

0x6e commented Jun 14, 2024 • edited Loading

tronical left a comment

Choose a reason for hiding this comment

0x6e commented Jun 14, 2024 •

edited

Loading