RFC — DtSh, shell-like interface with Devicetree

[dottspina]

RFC: DTSh: DTS file viewer with a shell-like command line interface

Introduction

Problem description

Prior to Zephyr, I've only approached Devicetree while looking through LKML or The Linux Kernel documentation.

And, quoting @mbolivar-nordic (October 2020):

Is your enhancement proposal related to a problem? Please
describe.

Devicetree is widely used by Zephyr, but it lacks convenient tooling
for development.

Describe the solution you'd like

A graphical devicetree viewer and editor, like 'guiconfig' for DTS.

I can agree that, when learning Zephyr use of Devicetree, I've personally felt the lack of a quick, simple tool to:

• see the final devicetree generated at build-time
• find nodes based on supported bus protocols, generated IRQs, or keywords like "BME680" or "PWM"
• access the relevant binding files
• generate simple figures to illustrate my note-taking

References:

• Improved user interface for devicetree (like guiconfig) #29295
• RFC - DTV (Device Tree Visualiser)

Proposed change

The proposed change is a new West command that opens a Devicetree in a shell-like command line interface:

• browse a devicetree through a hierarchical file system metaphor
• search for devices, bindings, buses or interrupts with flexible criteria
• filter, sort and format commands output
• generate simple documentation artifacts (text, HTML, SVG) by redirecting the output
  of commands to files
• rich Textual User Interface, command line auto-completion, command history, user themes

$ west build
$ west dtsh
dtsh (0.0.99): Shell-like interface with Devicetree
How to exit: q, or quit, or exit, or press Ctrl-D

/
> cd &flash_controller

/soc/flash-controller@4001e000
> find -E --also-known-as (image|storage).* --format NKd -T
                             Also Known As               Description
                             ───────────────────────────────────────────────────────────────────────────────────
flash-controller@4001e000    flash_controller            Nordic NVMC (Non-Volatile Memory Controller)
└── flash@0                  flash0                      Flash node
    └── partitions                                       This binding is used to describe fixed partitions…
        ├── partition@c000   image-0, slot0_partition    Each child node of the fixed-partitions node represents…
        ├── partition@82000  image-1, slot1_partition    Each child node of the fixed-partitions node represents…
        └── partition@f8000  storage, storage_partition  Each child node of the fixed-partitions node represents…

where:

• west dtsh: when invoked without any argument, the Devicetree shell will open the DTS file build/zephyr/zephyr.dts, and retrieve the bindings Zephyr has used at build-time from the CMake cache file build/CMakeCache.txt
• cd &flash_controller changes the current working directory from the devicetree's root to the node at /soc/flash-controller@4001e000, using its DTS label flash_controller
• find is with no surprise a shell command that will search for devices, bindings, buses, interrupts, etc
• here -E --also-known-as (image|storage).* will match nodes with a label or alias starting with image or storage; predicates like --with-reg-size >4k are also supported
• --format NKd: set the node output format to the columns "node Name,", "Also Known As" (all labels and aliases), and "description" (D is the "Depends-on" column)
• the descriptions, e.g. "Flash node", are actually hyperlinks that, when clicked, will open the corresponding binding files
• -T: list found nodes in tree-like format
• appending > doc/partitions.svg to the last command line would save the partitions tree to the file doc/partitions.svg, in SVG format

The considered use cases include:

• to help getting started with Devicetree: hierarchically or semantically explore a devicetree, contextually access binding files or the Devicetree specification, save figures to illustrate notes
• to have on hand a simple DTS file viewer: quickly check the enabled buses and connected devices or the RAM and Flash memory, get a first insight when debugging a Devicetree issue, document hardware configurations
• an extensible framework for prototyping TUI tools based on Zephyr's python-devicetree library

This proposal started with a Proof of Concept project: source code and documentation for this prototype are still available as the main branch of the DTSh repository.

The new code base in this RFC is also distributed as a stand-alone package you can install from PyPI without messing with your Zephyr wokspaces: for details, take a look the dtsh-next branch which mirrors this PR in the DTSh repository. Note that this version can't provide West integration: just replace west dtsh with dtsh.

Please refer to the attached DTSh User Guide (PDF) for the complete documentation of what this RFC implements and examples of use.

Detailed RFC

This RFC includes:

• the Devicetree Shell (DTSh) implementation in Python and its unit tests
• a West command for opening DTS files with DTSh
• documentation in reStructuredText format

Proposed change (Detailed)

This RFC wouldn't be review-able as a single commit: it is introduced as a series, where each commit adds a new feature or API, with its companion unit tests when they exist.

Once reviewed, we can squash or rearrange the whole thing according to the maintainers' preferences.

DTSh

DTSh is implemented in Python and is a Devicetree tool: it seems its natural location is zephyr/scripts/dts/dtsh, sibling to python-devicetree with which it's tightly coupled.

Source files are located in dtsh/src, and dtsh is the root Python package.

Unit tests are implemented with pytest, and located in dtsh/tests.
The res sub-directory contains test resource files.

Devicetree model

DTSh introduces its own model layer above edtlib.EDT.

Rationale:

• factorize DTSh interface with the edtlib API (python-devicetree)
• support the hierarchical file system metaphor at the model layer
• unified API for sorting and matching nodes
• write most unit tests at the model layer

Module		Unit tests
dtsh.dts	devicetree source definition	test_dtsh_dts
dtsh.model	devicetree model	test_dtsh_model
dtsh.modelutils	devicetree model helpers (e.g. sorters, criteria)	test_dtsh_modelutils

GNU Readline integration

Responsibilities:

• provide DTSh client code with a single entry-point API for GNU Readline integration
• isolate the completion logic (find matches) and view providers (display matches)
  from the Readline hooks machinery

Module		Unit tests
dtsh.rl	GNU readline integration API
dtsh.autocomp	base completion logic and view providers	test_dtsh_autocomp

Devicetree shell

Responsibilities:

• command line parser
• devicetree shell sessions setup
• helpers for implementing shell commands

Module		Unit tests
dtsh.io	base I/O streams for the devicetree shell	test_dtsh_io
dtsh.config	devicetree shell configuration	test_dtsh_config
dtsh.shell	devicetree shell	test_dtsh_shell
dtsh.shellutils	common command flags, arguments and parameters	test_dtsh_shellutils
dtsh.session	base devicetree shell session

rich TUI

DTSh's rich Textual User Interface is built on Texualize's rich library.

Module		Unit tests
dtsh.rich.theme	colors and styles	test_dtsh_theme
dtsh.rich.tui	base layouts
dtsh.rich.text	factories for styled text
dtsh.rich.modelview	factories for devicetree model views
dtsh.rich.svg	contents format for output redirection (SVG)	test_dtsh_rich_svg
dtsh.rich.io	rich I/O streams for the devicetree shell
dtsh.rich.autocomp	rich completion view providers
dtsh.rich.shellutils	command flags and arguments for formatted output	test_dtsh_rich_shellutils
dtsh.rich.session	rich devicetree shell session

Built-in Commands

Bellow are the currently implemented commands.

Module		Unit tests
dtsh.builtins.cd	cd: change the current working branch	test_dtsh_builtin_cd
dtsh.builtins.pwd	pwd: print path of current working branch.	test_dtsh_builtin_pwd
dtsh.builtins.ls	ls: list branch contents	test_dtsh_builtin_ls
dtsh.builtins.tree	tree: list branch contents in tree-like format	test_dtsh_builtin_tree
dtsh.builtins.find	find: search branches for nodes	test_dtsh_builtin_find
dtsh.builtins.alias	alias: list aliased nodes	test_dtsh_builtin_alias
dtsh.builtins.chosen	chosen: list chosen nodes	test_dtsh_builtin_chosen

West extension

West command implementation:

• class DTShell in zephyr/scripts/west_commands/dtshell.py
• packages dtsh and devicetree imported with os.sys.insert() statements

Added dtsh extension to West in zephyr/scripts/west-commands.yml.

Added DTSh run-time requirements to zephyr/scripts/requirements-base.txt:

• rich (GitHub, MIT License)
• macOS only, gnureadline (GitHub, GPL)

Added support for West completion to west-completion.bash and west-completion.zsh.

Documentation

The proposed change:

• adds a brief entry in zephyr-cmds.rst to document the West command
• that links to a detailed user guide (dtsh.rst) in the same directory

Dependencies

DTSh should not affect other components or needs changes in other areas.

But that's not true the other way around: changes in the Zephyr's hardware model may require changes in DTSh (see bellow).

Concerns and Unresolved Questions

Maintenance

This is a valid concern:

• the code base is sizable
• though Zephyr won't depend on DTSh in any way, DTSh is tightly coupled with Zephyr's hardware model and the companion edtlib API: this might introduce some additional maintenance for it to consistently follow changes

The everyday maintenance should be low, though:

• the Python module dtsh.model encapsulates all uses of the edtlib API: if breaking changes are introduced, DTSh will fail there
• the included unit tests should catch the most obvious regressions early, and help narrowing issues

The above observations come from my own experience (an example of breaking change was identifying top-level bindings with (compat, bus) tuples rather than just compatible strings).

More significant changes, like "Hardware model v2" or "Better support for multi-core AMP SoCs", could however involve more work, to either fix DTSh or add support for the new model.

Fortunately:

• such model changes don't happen every full moon night
• Zephyr seems to operate this kind of transitions cautiously: changes are discussed in advance, and introduced incrementally with some backward compatibility

Finally, I'll gladly continue myself contributing support, features and bug fixing in the long run.

Limitations

This RFC is not a complete Devicetree shell:

• there's no syntax to access an arbitrary property's value or description (binding): this will be the responsibility of a cat command, with a syntax like cat soc/timer$max-bit-width
• it misses a versatile man command, with manual pages for DTSh itself, but more importantly also for bindings and boards

I've already implemented these when experimenting with the PoC project: I know what to do, and how.

But this RFC is already sizable, and will require quite some work from its reviewers: I'd prefer we first agree on the initial design, implementation and documentation, then I'll push the missing commands with shortest individual PRs. This has been discussed and so far seems to be a consensus.

GNU Realine on Windows

Or the lack of.

For auto-completion, command history and key bindings, DTSh relies on the GNU Readline variant of the standard Python readline API.

Basically, there's no sane and straight forward way to build and distribute the GNU Readline library on non POSIX systems: on Windows, Python simply gave up and does no longer include the readline module with its standard library (see e.g. There's no readline module on Windows Python (cmd.Cmd) for some historical context).

The stand-alone module pyrealine3 cited in the above reference has been investigated as a work-around, but its API is incomplete, and its implementation too much coupled with cmd.Cmd, which we don't use. The project does not seem actively maintained.

To get a cross-platform user experience, I've also considered the pure Python Prompt Toolkit:

• what I've been able to achieve was clearly disappointing compared to binding native GNU Readline hooks to rich callbacks
• using both the Prompt Toolkit and Textualize's rich library was painful, and introduced other issues

Although DTSh may eventually work around this limitation, this is clearly not a priority at the moment.

As a consequence, the GNU Readline integration will very likely be disabled on Windows, resulting in a degraded user experience.

However, I am quite optimistic that this will not be considered a merge blocker: Zephyr seems to admit this kind of little (though very unfortunate here) difference between POSIX and non POSIX systems, e.g. west itself also supports auto-completion only with the Bash and Zsh shells.

User Guide Integration

The DTSh User Guide (dtsh.rst) is quite long, and contains sections, subsections, sub-subsections, etc.

Its integration with Zephyr's documentation is almost fine in HTML, and I'm not sure how to do better without messing with the :toctree:, which I prefer to avoid.

Its integration when generating PDF is more debatable (look at the attached PDF, concerns should be obvious):

• since the User Guide's content in the PDF document already starts at the "Second section heading level", proper styles quickly run out to highlight its own headings
• for the same reason, the TOC pane (typically shown at left in PDF viewers) will flatten the User Guide to a single heading level

The final PDF document is still legible, but I admit it's not optimal: I will gladly try any ideas for improvement.

Correctness

DTSh use cases include educational tools for newcomers to Zephyr Devicetree: if not correct, erroneous results (commands output) will do more harm than good.

Although correctness is the focus of unit tests, and code coverage for the core modules should be almost complete, these tests involve only a small subset of the possible configurations (MCU, SoC, boards, shields, peripherals).

I think we could define a minimum level of correctness as:

• DTSh should not fault when opening any valid Devicetree source (DTS and bindings), targeting any hardware configuration
• with these, all example commands found in the User Guide should consistently execute: important information specific to some configuration might be missing because unanticipated, but commands should not produce incorrect or misleading output

Security

The proposed change is not that hazardous:

• dtsh does not evaluate (eval()) any part of the user input
• dtsh does not pipe commands to the system interpreter (e.g. os.system() or os.popen()), nor create any kind of child process (e.g. subprocess.run())
• Devicetree source and binding files, CMake cache files, etc, are all opened read-only
• the writable user specific files (command history, custom configuration and theme files) are written to (and read from) a per-user application data directory located under the appropriate platform-dependent root (e.g. $XDG_CONFIG_HOME or %LOCALAPPDATA%)
• by default dtsh won't override any existing file (including when redirecting commands output)
• the West command itself is rather simple

Personal biases

Why a command line application ? Well, when you've done west build, you're already at the command line, and I think continuing from there with west dtsh, getting a different prompt but the same user interface paradigms and even key bindings (the base ones used by Zsh, GNU Bash, Emacs, GDB), is more ditrsaction-free than opening a GUI.

Beyond this personal biases, things like PyGObject or Qt for Python have nonetheless been considered, but:

• these would be huge requirements to add to Zephyr, and to the users' workspaces
• I wanted something simple and fast, easy to extend and to prototype with
• at the end of the day, I'm not sure a full GUI will be more user-friendly or informative than what is possible with TUI/CLI

Alternatives

There doesn't seem to be many DTS file viewers.

Nordic Semiconductor's distributes the nRF DeviceTree extension for VS Code, which looks interesting but does not address the initial problem (as @hongshui3000 also pointed out):

• it's not Open Source, and seems to assume you're using the nRF Connect SDK, not Zephyr
• VS Code with the nRF Connect Extension Pack is not really a simple tool everyone is supposed to have on hand (or to work with)

[hongshui3000]

I think VS Code extension: nRF DeviceTree is a good graphical tool. But it's not open source. Not particularly friendly to non-nRF devices

[mbolivar-ampere]

Thanks for this. I think it's a great idea and I especially appreciate the attention to detail you put into the RFC as well as volunteering maintenance. Consider me onboard with this idea. I'll give it some testing and review.

[mbolivar-ampere]

Some unboxing comments:

Python provided by Homebrew or Fink: should also work OOTB

I can't get tab completion working as I expect using the homebrew python 3.11. From the prompt:

/
❭ 

I press TAB and expect commands to be completed. But instead I see a tab character is inserted. Is that expected? Here is my python info (using a virtual environment):

(devel) mbolivar@PDX-FY74HK3HFV zephyr % python -V
Python 3.11.4
(devel) mbolivar@PDX-FY74HK3HFV zephyr % echo $(readlink $(which python))
/opt/homebrew/opt/python@3.11/bin/python3.11

I can import readline from the python REPL with no error. I did a pip install -r zp/zephyr/scripts/dts/dtsh/requirements.txt before running west dtsh.

The command parser also doesn't seem to be stripping leading whitespace:

/
❭     q
dtsh: command not found: q

/
❭ q
bye.

this isn't like what you'd expect from a shell in my opinion -- I would expect both of those to quit.

[dottspina]

@MaureenHelm , @decsny , thanks again for taking some time with this.

This encouraged me to update the PR (if west dtsh were to find its way upstream into zephyr-rtos/zephyr or as an optional module, a code review would be required in both cases):

• rebased onto Zephyr latest (commit f05d16d)
• rebased onto DTSh latest (commit ae5df504c5c): well, actually not something like git rebase, contents of the dtsh directory in this PR (zephyr/scripts/dts/dtsh) can't mirror DTSh, but at least the source code is the same (aside from the West command integration, obviously), and unit tests pass wherever the dtsh directory comes from (DTSh project or PR); efforts have also been made to share the documentation (reST files) when appropriate

Backporting changes from the DTSh project to the PR should be easier now, so that we can review this, introducing changes only in response to the process, and sync contents later on, when we know where we go, or not.

A few things that might help you get an idea of the current DTSh status:

• the Handbook: most things apply, just think $ west dtsh instead of $ dtsh, obviously
• DTSh 0.2.3 (PyPI): pip install dtsh==0.2.3 (recommended to skim through the Getting Started Guide which covers this approach)

I'll update the RFC description asap (minor changes).

Thanks.

Note: it seems that the CI check "Scancode / Scan code for licenses" is stricter (regarding shell files ?) than scripts/ci/check_compliance.py and scripts/checkpatch.pl, is this plausible ?

[fgrandel]

@dottspina Just stumbled upon this RFC. This is a great contribution and solves a real problem. I love how thoroughly this RFC is written and how carefully you evaluate and argue alternatives and potential issues. I'll try to help as much as I can in reviewing. Is there a way to contact you on Zephyr's Discord if I have questions, so that we can keep comment noise on this RFC focused on results rather than detailed discussions?