Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RFC — DtSh, shell-like interface with Devicetree #59863

Open
wants to merge 33 commits into
base: main
Choose a base branch
from
Open

RFC — DtSh, shell-like interface with Devicetree #59863

wants to merge 33 commits into from

Commits on Aug 27, 2024

  1. [dtsh 1/33] ci: RFC project setup 

    Setup RFC root directory as ZEPHYR_BASE/scripts/dts/dtsh.

Includes project artifacts useful for reviewing and hacking
the RFC (developer-only, and mostly personal biases or needs).

Signed-off-by: Christophe Dufaza <chris@openmarl.org>
    @dottspina
    dottspina committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    727f4b3 View commit details
    Browse the repository at this point in the history

  2. [dtsh 2/33] dts: devicetree source definition 

    A devicetree source is fully defined by:

- a DTS file in Devicetree source format (DTSpec 6)
- all of the YAML binding files the DTS recursively depends on
- the Devicetree Specification

This module may rely on cached CMake variables to locate
the binding files the DTS file was actually generated with.
The herein CMake cache reader implementation is adapted from
the zcmake.py module in zephyr/scripts/west_commands.

This module eventually introduces a YAML file system API
that should cover the devicetree shell needs:

- access the DT binding files with their base name
- recursively access YAML-included bindings
- the name2path semantic expected by edtlib.Binding ctor

Unit tests and examples: tests/test_dtsh_dts.py

Signed-off-by: Christophe Dufaza <chris@openmarl.org>
    @dottspina
    dottspina committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    68d9ae9 View commit details
    Browse the repository at this point in the history

  3. [dtsh 3/33] model: devicetree model 

    Rationale:

- factorize the dtsh interface with edtlib (aka python-devicetree)
- support the hierarchical file system metaphor at the model layer
- unified API for sorting and matching nodes
- provide model objects (nodes, bindings, etc) with identity, equality
  and a default order-by semantic
- write most dtsh unit tests at the model layer

Unit tests and examples: tests/test_dtsh_model.py

Signed-off-by: Christophe Dufaza <chris@openmarl.org>
    @dottspina
    dottspina committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    f12293d View commit details
    Browse the repository at this point in the history

  4. [dtsh 4/33] modelutils: devicetree model helpers 

    - Match nodes with text based criteria (implement DTNodeTextCriterion)
- Match nodes with integer based criteria (implement DTNodeIntCriterion)
- Sort nodes (implement DTNodeSorter)
- Arbitrary virtual devicetree (DTWalkableComb)
- Factory for string representations of DTS types

Unit tests and examples: tests/test_dtsh_modelutils.py

Signed-off-by: Christophe Dufaza <chris@openmarl.org>
    @dottspina
    dottspina committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    ed48d5a View commit details
    Browse the repository at this point in the history

  5. [dtsh 5/33] config: devicetree shell configuration and data 

    User-specific configuration and data are stored in a
platform-dependent directory.

DTSh is configured by simple INI files:

- the bundled configuration file which sets the default configuration
- an optional user's configuration file which customizes the defaults

The command history file (if GNU readline support is enabled)
is loaded from and stored into the same directory.

Unit tests and examples: tests/test_dtsh_config.py

Signed-off-by: Christophe Dufaza <chris@openmarl.org>
    @dottspina
    dottspina committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    4ede917 View commit details
    Browse the repository at this point in the history

  6. [dtsh 6/33] io: I/O streams for devicetree shells 

    - stdout: DTShOutput
- base input stream (empty command source): DTShInput
- base terminal API: DTShVT
- parsed command redirection: DTShRedirect
- command redirection to file: DTShOutputFile
- batch command sources: DTShInputFile, DTShInputCmds

Unit tests and examples: tests/test_dtsh_io.py

Signed-off-by: Christophe Dufaza <chris@openmarl.org>
    @dottspina
    dottspina committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    c5a3c59 View commit details
    Browse the repository at this point in the history

  7. [dtsh 7/33] rl: GNU Readline integration 

    Rationale:
- provide DTSh client code with a single entry-point API for
  GNU Readline integration
- isolate the completion logic (find matches) and view providers
  (display matches) from the readline hooks machinery

The GNU Readline integration with Python is initialized:
- with the standalone Python module gnureadline if installed: this
  is likely necessary on macOS, where readline is typically
  backed by libedit instead of libreadline
- with the readline module of the Python standard library
  on other POSIX systems

On windows, the readline support will likely be disabled.

Signed-off-by: Christophe Dufaza <chris@openmarl.org>
    @dottspina
    dottspina committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    359be64 View commit details
    Browse the repository at this point in the history

  8. [dtsh 8/33] shell: the devicetree shell 

    Shell-like interface with a devicetree model:

- parse command lines into commands, arguments and parameters
- walk the devicetree through a hierarchical file-system metaphor
  (current working branch, relative paths and path references,
  Devicetree labels)
- match nodes with flexible criteria
- sort and filter nodes

Unit tests and examples: tests/test_dtsh_shell.py

Signed-off-by: Christophe Dufaza <chris@openmarl.org>
    @dottspina
    dottspina committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    7e0d9a9 View commit details
    Browse the repository at this point in the history

  9. [dtsh 9/33] autocomp: completion logic and base display callbacks 

    Auto-completion logic and base display callbacks
for GNU Readline integration.

Unit tests and examples: tests/test_dtsh_autocomp.py

Signed-off-by: Christophe Dufaza <chris@openmarl.org>
    @dottspina
    dottspina committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    3d26961 View commit details
    Browse the repository at this point in the history

  10. [dtsh 10/33] shellutils: devicetree shell helpers 

    Flags, arguments and parameters for DTSh commands.

Unit tests and examples: tests/test_dtsh_shellutils.py

Signed-off-by: Christophe Dufaza <chris@openmarl.org>
    @dottspina
    dottspina committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    91f6c9f View commit details
    Browse the repository at this point in the history

  11. [dtsh 11/33] rich: theme: devicetree shell theme 

    Devicetree shell theme (aka rich styles).

A theme is a collection of styles, each of which is consistently used
to represent a type of information: e.g. by default compatible strings
are always green, and things that behave like symbolic
links (e.g. aliases) are all in italic.

Theme files are simple INI files:

- the bundled theme file which sets the default appearance
- an optional user's theme file which customizes the defaults

Unit tests and examples: tests/test_dtsh_theme.py

Signed-off-by: Christophe Dufaza <chris@openmarl.org>
    @dottspina
    dottspina committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    ac100bd View commit details
    Browse the repository at this point in the history

  12. [dtsh 12/33] rich: tui: rich console protocol 

    Base view definitions compatible
with the rich console protocol, __rich__().

Signed-off-by: Christophe Dufaza <chris@openmarl.org>
    @dottspina
    dottspina committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    13c92e0 View commit details
    Browse the repository at this point in the history

  13. [dtsh 13/33] rich: text: text view factories and other helpers 

    Factories for rich text views, and miscellaneous text related helpers.

Signed-off-by: Christophe Dufaza <chris@openmarl.org>
    @dottspina
    dottspina committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    716350d View commit details
    Browse the repository at this point in the history

  14. [dtsh 14/33] rich: modelview: model to views 

    Stateless factories of base Devicetree model elements.

Context-aware views of DT nodes.

Signed-off-by: Christophe Dufaza <chris@openmarl.org>
    @dottspina
    dottspina committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    17ad90b View commit details
    Browse the repository at this point in the history

  15. [dtsh 15/33] rich: svg: contents format for output redirection 

    SVG contents format for commands output redirection.

Rationale:
- the SVG documents we can generate directly with the
  Textualize's rich library API do not really suit
  the DTSh use case
- we need an additional abstraction layer to properly
  support the "append" mode

Signed-off-by: Christophe Dufaza <chris@openmarl.org>
    @dottspina
    dottspina committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    5d0cfdc View commit details
    Browse the repository at this point in the history

  16. [dtsh 16/33] rich: io: rich I/O streams for devicetree shells 

    - base rich VT (colors and styles)
- rich VT with batch commands support
- rich redirection streams for SVG and HTML

Signed-off-by: Christophe Dufaza <chris@openmarl.org>
    @dottspina
    dottspina committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    a4b94c2 View commit details
    Browse the repository at this point in the history

  17. [dtsh 17/33] rich: autocomp: rich auto-completion helper 

    Rich display callback for GNU readline integration

Style candidates based on completion context.

Signed-off-by: Christophe Dufaza <chris@openmarl.org>
    @dottspina
    dottspina committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    0fd2da1 View commit details
    Browse the repository at this point in the history

  18. [dtsh 18/33] rich: shellutils: helpers for formatted output 

    Command options to configure formatted outputs.

Base command with boilerplate code to support
formatted output.

Unit tests and examples: tests/test_dtsh_rich_shellutils.py

Signed-off-by: Christophe Dufaza <chris@openmarl.org>
    @dottspina
    dottspina committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    c457c6a View commit details
    Browse the repository at this point in the history

  19. [dtsh 19/33] builtin: pwd: print path of current working branch 

    Unit tests and examples: tests/test_dtsh_builtin_pwd.py

Signed-off-by: Christophe Dufaza <chris@openmarl.org>
    @dottspina
    dottspina committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    8a48ce9 View commit details
    Browse the repository at this point in the history

  20. [dtsh 20/33] builtin: cd: change the current working branch 

    Unit tests and examples: tests/test_dtsh_builtin_cd.py

Signed-off-by: Christophe Dufaza <chris@openmarl.org>
    @dottspina
    dottspina committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    eff1d3b View commit details
    Browse the repository at this point in the history

  21. [dtsh 21/33] builtin: ls: list branch contents 

    Unit tests and examples: tests/test_dtsh_builtin_ls.py

Signed-off-by: Christophe Dufaza <chris@openmarl.org>
    @dottspina
    dottspina committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    1ab0e86 View commit details
    Browse the repository at this point in the history

  22. [dtsh 22/33] builtin: tree: list branches in tree-like format 

    Unit tests and examples: tests/test_dtsh_builtin_tree.py

Signed-off-by: Christophe Dufaza <chris@openmarl.org>
    @dottspina
    dottspina committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    d7e856e View commit details
    Browse the repository at this point in the history

  23. [dtsh 23/33] builtin: find: search branches for nodes 

    Search for nodes with pre-defined criteria.

Unit tests and examples: tests/test_dtsh_builtin_find.py

Signed-off-by: Christophe Dufaza <chris@openmarl.org>
    @dottspina
    dottspina committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    12f1846 View commit details
    Browse the repository at this point in the history

  24. [dtsh 24/33] builtin: alias: list aliased nodes 

    Unit tests and examples: tests/test_dtsh_builtin_alias.py

Signed-off-by: Christophe Dufaza <chris@openmarl.org>
    @dottspina
    dottspina committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    fb39c32 View commit details
    Browse the repository at this point in the history

  25. [dtsh 25/33] builtin: chosen: list chosen nodes 

    Unit tests and examples: tests/test_dtsh_builtin_chosen.py

Signed-off-by: Christophe Dufaza <chris@openmarl.org>
    @dottspina
    dottspina committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    31928cf View commit details
    Browse the repository at this point in the history

  26. [dtsh 26/33] builtin: cat: concat and output info about a node 

    Concat and output info about a node and its properties.

Unit tests and examples: tests/test_dtsh_builtin_cat.py

Signed-off-by: Christophe Dufaza <chris@openmarl.org>
    @dottspina
    dottspina committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    ab4af26 View commit details
    Browse the repository at this point in the history

  27. [dtsh 27/33] builtin: uname: print system information 

    Print system information (kernel, board, SoC).

Signed-off-by: Christophe Dufaza <chris@openmarl.org>
    @dottspina
    dottspina committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    8ac0dd4 View commit details
    Browse the repository at this point in the history

  28. [dtsh 28/33] session: base devicetree shell session 

    A session binds a devicetree shell and I/O streams to:
- execute batch commands
- and/or run an interactive loop

Signed-off-by: Christophe Dufaza <chris@openmarl.org>
    @dottspina
    dottspina committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    84775d1 View commit details
    Browse the repository at this point in the history

  29. [dtsh 29/33] rich: session: rich devicetree shell session 

    Extend the base session with a few rich TUI elements
and support for SVG and HTML command output redirection formats.

Signed-off-by: Christophe Dufaza <chris@openmarl.org>
    @dottspina
    dottspina committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    853e4d3 View commit details
    Browse the repository at this point in the history

  30. [dtsh 30/33] cli: devicetree shell CLI 

    Devicetree shell CLI.

Rationale:
- factorize command line parser initialization
  (options and arguments definitions)
- factorize the boilerplate code needed to bootstrap a DTSh session
 (CLI or batch, loading additional theme and preferences files, etc)
- provide an API suitable for both standalone installations (raw CLI)
  or integration as a West extension

The API then defines two distinct simple usages:
- a standalone mode (raw CLI entry point): DTShCli is then responsible
  for initializing a default CLI parser, and actually parsing the
  command line arguments when entering DTShCli.run()
- a passive mode: some external code (e.g. a WestCommand) is responsible
  for configuring the parser it manages with DTShArgvParser.init(),
  and actually parsing the command line arguments needed
  to call DTShCli.run()

Signed-off-by: Christophe Dufaza <chris@openmarl.org>
    @dottspina
    dottspina committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    36a5a91 View commit details
    Browse the repository at this point in the history

  31. [dtsh 31/33] west: dtsh integration as Zephyr West extension 

    - West command: class DTShell in scripts/west_commands/dtshell.py
- dtsh extension to West in west-commands.yml.
- support for completion west-completion.{bash,zsh}

Signed-off-by: Christophe Dufaza <chris@openmarl.org>
    @dottspina
    dottspina committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    6620471 View commit details
    Browse the repository at this point in the history

  32. [dtsh 32/33] tests: initial unit test coverage 

    Initial unit test coverage:
- core (model, shell, sessions): mostly complete
- commands (builtins): basics (options, parameters)
  for all commands but "uname"
- views (rich): actual unit testing would require
  some framework to compare the created views
  to the expected output; we at least check that
  all views build for all nodes in the test model

Signed-off-by: Christophe Dufaza <chris@openmarl.org>
    @dottspina
    dottspina committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    1e8b87c View commit details
    Browse the repository at this point in the history

  33. [dtsh 33/33] doc: West command documentation, handbook 

    Document "west dtsh" in "Additional Zephyr extension commands".

Document DTSh itself in the Handbook.

Signed-off-by: Christophe Dufaza <chris@openmarl.org>
    @dottspina
    dottspina committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    d86425f View commit details
    Browse the repository at this point in the history