-
Notifications
You must be signed in to change notification settings - Fork 98
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Testing/CI/Release For Embedded Targets #16
Comments
I'm building with Xargo on Travis for Cortex-M4 but I don't have a strategy for test as yet (so maybe that counts as 'trivial', I don't know). At work, we would generally hook up target hardware to our CI system to flash binaries and run tests on target, but this is a personal project for now. |
I have been working in this space 😄. I really liked While building smoke (a repository that's meant to test the nightly releases of libstd against as many targets as possible (though it's not exactly running on a nightly basis right now)), I figured out how to transparently run cross compiled programs under QEMU. By transparent, I meant that I'm also the author of rust-everywhere which is a CI "template" that produces binary releases of a crate for tier 1 targets (Linux, OSX and Windows) and uploads them to GitHub releases. So, lately I have been trying to merge these three concepts: one Docker image per target, transparent cross testing using QEMU and deploys (binary releases). The result so far is trust. Which is, right now, also a CI "template" that uses Docker and QEMU to test and make deploys for 20+ different targets (including the new Trust is already functional. I have been using it as a CI template in my latest embedded projects to test my crates for the So, yeah. I wouldn't anyone to use trust until there's an upgrade mechanism in place.
Pure Rust libraries that (want to) optimize their implementations on some architectures using assembly should also test on different targets. Libraries in this category are compiler-builtins and m. If you are using repr(C) or |
@jamesmunns wrote an excellent blog post about the different ways to do CI testing in the embedded world. The approaches listed there are: CI BuildBasically run I think this method is well supported today thanks to Non-Host TestingBasically You have structure your embedded crate correctly so that unit tests can be run on the host. This is mainly for testing logic that doesn't involve target device features / hardware as those are not present on Travis builders / your laptop. This approach is not "high fidelity" as there are some differences between the target device and the Travis builder: for example, the size of Host TestingBasically As you may know the Simulated Host TestingBasically Again Do note that QEMU doesn't emulate device hardware like I2C and PWM so you can't test that functionality using this method. Hardware In the Loop (HIL)This would be some higher level kind of integration testing where a PC sends some input to the target device, the target device responds with some output and the PC verifies that the output is indeed correct. AFAIK, no solution for this exists in the ecosystem. I think that both CI builds and non-host testing are well supported today and can be easily be implemented using Travis CI. The utest framework can be used for host testing and simulated host testing but is it in good shape enough? Could it be improved further? How does it compare to other implementations used in the C/C++ world? I'd like more people to give it a try, specially in the host testing deparment. Does anyone has any idea of how a HIL framework would look like for Rust? Or does anyone know of a well established HIL framework that we could port to Rust? |
Most of my work products achieve HIL testing by exposing the functional API at each level of the stack through a command line harness. This can compile for either host and embedded target. You can drive the interface manually or with an automated tool. |
I recently released Bobbin CLI, which includes a simple text-based serial console based test runner. I experimented with a full-fledged two-way COBS encoded TLV-style protocol, but eventually decided that I wanted something much, much simpler.
The test runner will exit with return code 1 if there is a delay of more than 5 seconds between lines or 15 seconds to complete the entire test. In the future these timeouts will be configurable. This system doesn't currently handle tests that are intentionally supposed to trap or panic, but I can think of a few ways to make it possible with some help from a watchdog timer. Currently this is only implemented for devices with serial consoles, but it should be straightforward to make this work using SWO for targets and debuggers that support it. |
@japaric do you know of any examples of anyone doing this? I’m very interested in your work on utest, but I’m currently interested in how to organize a crate so that I can run whatever tests I can on host. In C++ I essentially have 2 makefiles. One for host and one for target. Any idea of how to accomplish something similar with Rust? |
Closing this as part of 2024 triage. As of now, the charter of the WG is to focus on foundational crates and functionality, rather than develop solutions for all use cases. However, as of today there are the following existing solutions, and folks finding this issue are suggested to contribute to one of the following projects:
If you think this was closed incorrectly, please leave a comment and we can revisit this decision in the next weekly WG meeting on Matrix. |
Problem
Embedded targets typically do not target the same architectures that are readily available on CI services like Travis and developers machines, yet we would still like for CI to be able to build our software for alternative architectures and, ideally, run tests on that architecture.
As an example, let's consider another project I am a maintainer on, nix-rust/nix, which although not strictly focused on embedded is likely to be needed on many embedded systems (it is a dependency of mio and many other crates as it provides a set of safer APIs on top of libc which may not be present in
std
). For this project, we want to do the following:Matrixed with these additional things for each of the above:
Project Case Studies
Currently, there are several projects that implement their own solutions to this problem (to varying degrees of success) and a few projects which exist to help aid developers who are seeking to build/test for several different platforms.
rust-lang/libc
The libc crate is built for and runs tests against a number of different targets including several which are not yet officially supported. The libc crate contains a CI Directory which provides an overview of the strategy it uses for doing cross-build/testing.
This boils down to the following (ignoring platforms like Windows/OSX that are not really relevant for embedded):
TARGET
variable and the desired rust version is specified with therust
variable in the travis matrix. Linux is used for the host OS. E.g.$TARGET
:docker build
to create a docker image using the Dockerfile inci/docker/$TARGET
. E.g. https://github.com/rust-lang/libc/blob/master/ci/docker/arm-unknown-linux-gnueabihf/Dockerfile. The docker images contain all non-rust dependencies that are required to build and test for$TARGET
. Typically, this is gcc/libc (for the libc crate) andqemu-user
.docker run
to launch a new container from the previously built image and executes the run.sh script-v
rustc --print sysroot:/rust:ro
) is shared with the container at /rustQEMU
environment variable is specified, the QEMU image is fetched and and a bunch of rigamarole and setup is performed in order to set things up. Finally, QEMU is run and the output is parsed to determine success. The only emulated machine currently is x86_64.libc-test
binary (cross compilation of this binary for the target is done on the host.Things work well, but there is a moderate amount of complexity. Effort that is done to improve testing for the libc crate do not directly help out other projects which might want the same enhancements.
Major contributors here have been @alexchrichton, @japaric, @semarie
nix-rust/nix
This nix-rust/nix crate is based off of the work done in libc in an earlier version and has diverged some since then. This work was originally done by yours truly.
This work was done by @posborne based on libc.
japaric/smoke
I haven't looked at this one much but @japaric has some relevant experience. Appears to use some combination of Docker/QEMU for build/testing. Dockerfile appears to be monolithic.
Others?
Looking for feedback on other projects that are doing a non-trivial amount with doing cross-build/test within the Rust ecosystem. MCU targets would be appreciated in addition to the above projects which focus on targets with an OS.
Ecosystem Projects
rust-embedded/docker-rust-cross
The goal of this repository was to provide the Docker images (a common theme for this work) in order to perform cross-compilation and cross-test for non-host architectures. To date, I have not been diligent about keeping these images up-to-date with each Rust release.
The goal is to have a common repository of Docker images that can be reused across projects like libc/nix/others so each project doesn't need to go in the weeds on that front.
@posborne is the maintainer of this project.
japaric/rust-everywhere
Rust-everywhere seeks to make it easier to cross-compile crates for other targets and publish binaries. It does not appear to help out with running tests on foreign architectures, although this is something that is likely to be desirable for projects targeting other architectures.
Final Note
For many libraries that do not do FFI (especially libc/kernel FFI), the importance of actually running tests on the target architecture is probably not as strong. Within embedded, however, there seems to be a much greater chance that using APIs that may not already have nice interfaces is increased. As such, I feel this forum is still an appropriate place to discuss how we want to handle this problem moving forward.
The text was updated successfully, but these errors were encountered: