These release notes are complemented by two other documents:
- 🪵 Changelog for a structured list of additions, fixes, changes, and removals.
- 💔 Breaking Changes for a list of impacts to existing Ceedling projects.
This Ceedling release is probably the most significant since the project was first posted to SourceForge in 2009.
Ceedling now runs in Ruby 3. Builds can now run much faster than previous versions because of parallelized tasks. In test suites, header file search paths, code defines, and tool run flags are now customizable per test executable. Ceedling now also offers integrated debugging options to find the cause of crashing tests.
- See all the Highlights below for an overview of everything in this big release.
- See the Changelog for a detailed list of new features, fixes, and changes in this release.
- We’ve included below a project configuration cheatsheet for those already familiar with Ceedling that want to understand the various configuration changes.
Ahoy! There be plenty o’ breaking changes ahead too, mateys! Arrr…
A HUGE Thanks to the ThrowTheSwitch.org community, for continuing to use, critique, and contribute to these tools. We're making the C world a better place and we appreciate all of you!
We'd like to make some quick shout-outs to some especially helpful contributions. THANK YOU!
-- Mark VanderVoord & Machael Karlesky, ThrowTheSwitch.org Project Maintainers
- ThingamaByte, LLC - For continuing to nurture these projects and community with so much of their time.
- Kamstrup, A/S - For sponsoring and testing Ceedling's new parallel build/test support
- Fraunhofer Institute for Integrated Systems and Device Technology IISB - For also sponsoring and testing Ceedling's new parallel build/test support
- Peter Membrey - For sponsoring, helping to plan, and validating Ceedling's new dependencies plugin
These individuals contributed significant features, bugfixes, and improvements. This list was generated by git, so if you feel you should be here, you're probably right. Please let us know and we're very sorry to have missed you!
- Dom Postorivo
- Cezary Gapinski
- Aaron Schlicht
- Tim Bates
- Patrick Junger
- Austin Glaser
- Łukasz Żegliński
- Anthony Male
- Peter Membrey
- Laurens Miers
- Alejandro Rosso
- Tony Webb
- Greg Williams
- John Van Enk
- Job Vranish
These individuals have been a huge help in answering questions and guiding newcomers on our forums. Thanks for making this a welcoming community!
- @Letme
- @dpostorivo
- @swaldhoer
- @CezaryGapinski
- @aschlicht
Hiroaki Yamazoe, Lucas Becker, Rafael Campos Las Heras, Scott Vokes, Zane D. Purvis, Данила Вальковец, Aaron Schlicht, Carl-Oskar Larsson, Javier Tiá, Jesper L. Nielsen, MrRedKite, Nik Krause, Rasmus Melchior Jacobsen, serjche, Andrei Korigodskii, Eder Clayton, Felipe Balbi, Hannes Bachl, Mike D. Lowis, Peter Horn, Rasmus Uth Pedersen, Simon Grossenbacher, AlexandreMarkus, André Draszik, Aurelien CHAPPUIS, Austin Yates, Ben Wainwright, Christopher Cichiwskyj, Crt Mori, Horacio Mijail Antón Quiles, John Hutchinson, Joseph Gaudet, Julien Peyregne, KGons, Kalle Møller, Peter Kempter, Luca Cavalli, Maksim Chichikalov, Marcelo Jo, Matt Chernosky, Niall Cooling, Olivier C. Larocque, Patrick Little, Richard Eklycke, Serjche, Spencer Russell, Stavros Vagionitis, Steven Huang, Toby Mole, Tom Hotston, Yuanqing Liu, afacotti, ccarrizosa, diachini, Steven Willard
The following is not a complete project configuration. But, for those already familiar with Ceedling, this cheatsheet illustrates some of the important changes in this latest release of Ceedling through the lens of a project configuration.
To be clear, more has changed than what is referenced in this YAML blurb. Most notably:
- Ceedling’s command line is more robust and conforms to common CLI conventions.
- Various plugins have grown in functionality and changed in name and configuration convention.
- Build directive macros are available that provide build customization abilities through their use in your test files.
# Mixins are an all new feature that allow you to easily merge alternate project configurations.
# Mixins replace a variety of other features, namely the :import project file section and option: command line task.
# A frequent need is a base project configuration complemented by variants of projects for different flavors of the same codebase. Mixins provide the features to support these kinds of needs.
# Mixins have few limitations as compared to the features Mixins replace.
# Mixins can be merged from within your project configuration file (shown here), from paths in environment variables, and from command line --mixin path switches.
# ---------------------------
:mixins:
:enabled: # :enabled list supports names and filepaths
- enabled # Look for enabled.yml in load paths and merge if found
- my/mixin/cfg.yml # A full path to a configuration file to merge
:load_paths: # Load paths to search for mixins specified by name only in :enabled
- support/mixins
# Importing project configuration files with :import is superseded by the more capable Mixins (above).
# ---------------------------
# :import:
# - path/to/config.yml
:project:
# Ceedling is now multi-threaded for speedy builds.
# The default is a single thread (setting of 1). :auto tells Ceedling to query your system and make an educated guess on a number of threads to use.
# Threading is broken into two categories to provide for controlling threading where build tooling and test executable tooling have different restrictions.
# ---------------------------
:compile_threads: :auto
:test_threads: :auto
# Ceedling's preprocessing abilities have been totally revamped to fix bugs, eliminate complexity, and improve results.
# Preprocessing can now be selectively applied to test files and to mockable headers using options :none, :mocks, :tests, or :all.
# ---------------------------
:use_test_preprocessor: :all
# The following configuration options have been deprecated.
# These are no longer available as the underlying functionality has been reimplemented with no need to configure it.
# If these settings remain in your configuration file, they are harmless but do zilch for you.
# ---------------------------
# :use_deep_dependencies
# :generate_deep_dependencies
# :auto_link_deep_dependencies
# Backtrace is an all new feature in Ceedling.
# When enabled (default is :simple), backtrace figures out which test case in a crashed test executable is exercising the bug causing you grief.
# If the :gdb option is enabled (and the GNU debugger is installed), Ceedling will provide you the trace to the line of code causing a test case to crash.
# ---------------------------
:use_backtrace: :simple
# Complemeneting its existing abilities to build assembly code as part of a release artifact, Ceedling can now also incorporate assembly code into test builds.
# See CeedlingPacket for full details on how to make use of this feature after enabling it.
# In short, your assembly code will need to be findable in your project paths, and you will need to use the new test directive macro TEST_SOURCE_FILE() inside your test files to tell Ceedling which assembly file to include in the respective test executable build.
:test_build:
:use_assembly: TRUE
# Ceedling executables are now built as self-contained mini-projects.
# You can now define symbols for a release build and each test executable build.
# Symbols defined for a test executable are applied during compilation for each component of the executable.
# The :defines section supports several syntaxes.
# - Sophisticated matchers are available for test executable builds (shown here).
# - Simple lists to apply flags to all files in a build step are supported for release builds (only option) and test builds.
# See Ceedling Packet for details.
# ---------------------------
:defines:
:test:
:*: # Wildcard: Add '-DA' for compilation of all files for all tests
- A
:Model: # Substring: Add '-DCHOO' for compilation of all files of any test with 'Model' in its name
- CHOO
:/M(ain|odel)/: # Regex: Add '-DBLESS_YOU' for all files of any test with 'Main' or 'Model' in its name
- BLESS_YOU
:Comms*Model: # Wildcard: Add '-DTHANKS' for all files of any test that have zero or more characters
- THANKS # between 'Comms' and 'Model'
:release:
- FEATURE_X=ON # Add these two symbols to compilation of all release C files (:test supports this syntax too)
- PRODUCT_CONFIG_C
# Ceedling executables are now built as self-contained mini-projects.
# Flags can be specified for each build step and for each component of a build step.
# The :flags section supports several syntaxes.
# - Sophisticated matchers are available for test executable builds (shown here).
# - Simple lists to apply flags to all files in a build step are supported for release builds (only option) and test builds.
# See Ceedling Packet for details.
# ---------------------------
:flags:
:test:
:compile:
:*: # Wildcard: Add '-foo' for all files for all tests
- -foo
:Model: # Substring: Add '-Wall' for all files of any test with 'Model' in its name
- -Wall
:/M(ain|odel)/: # Regex: Add 🏴☠️ flag for all files of any test with 'Main' or 'Model' in its name
- -🏴☠️
:Comms*Model:
- --freak # Wildcard: Add your `--freak` flag for all files of any test name with zero or more
# characters between 'Comms' and 'Model'
:release:
:compile:
- -std=c99 # Add `-std=c99` to compilation of all release build C files (:test supports this syntax too)
# Ceedling’s Unity configuration now properly supports test executable builds for Unity's parameterized test cases.
# Previously a handful of settings were required throughout a project configuration to successfully use these abilities.
# ---------------------------
:unity:
:use_param_tests: TRUE
Ceedling now runs in Ruby3. This latest version of Ceedling is not backwards compatible with earlier versions of Ruby.
Previously, Ceedling builds were depth-first and limited to a single line of execution. This limitation was an artifact of how Ceedling was architected and relying on general purpose Rake for the build pipeline. Rake does, in fact, support multi-threaded builds, but, Ceedling was unable to take advantage of this. As such, builds were limited to a single line of execution no matter how many CPU resources were available.
Ceedling 1.0.0 introduces a new build pipeline that batches build steps breadth-first. This means all test preprocessor steps, all compilation steps, all linking steps, etc. can benefit from concurrent and parallel execution. This speedup applies to both test suite and release builds.
In previous versions of Ceedling each test executable was built with essentially the same global configuration. In the case of #defines and tool command line flags, individual files could be handled differently, but configuring Ceedling for doing so for all the files in any one test executable was tedious and error prone.
Now Ceedling builds each test executable as a mini project where header file search paths, compilation #define symbols, and tool flags can be specified per test executable. That is, each file that ultimately comprises a test executable is handled with the same configuration as the other files that make up that test executable.
Now you can have tests with quite different configurations and behaviors. Two tests need different mocks of the same header file? No problem. You want to test the same source file two different ways? We got you.
The following new features (discussed in later sections) contribute to this new ability:
TEST_INCLUDE_PATH(...). This build directive macro can be used within a test file to tell Ceedling which header search paths should be used during compilation. These paths are only used for compiling the files that comprise that test executable.:defineshandling.#defines are now specified for the compilation of all modules comprising a test executable. Matching is only against test file names but now includes wildcard and regular expression options.:flagshandling. Flags (e.g.-std=c99) are now specified for the build steps — preprocessing, compilation, and linking — of all modules comprising a test executable. Matching is only against test file names and now includes more sensible and robust wildcard and regular expression options.
Ever wanted to smoosh in some extra configuration selectively? Let’s say you have different build scenarios and you'd like to run different variations of your project for them. Maybe you have core configuration that is common to all those scenarios. Previous versions of Ceedling included a handful of features that partially met these sorts of needs.
All such features have been superseded by Mixins. Mixins are simply additional YAML that gets merged into you base project configuration. However, Mixins provide several key improvements over previous features:
- Mixins can be as little or as much configuration as you want. You could push all your configuration into mixins with a base project file including nothing but a
:mixinssection. - Mixins can be specified in your project configuration, via environment variables, and from the command line. A clear order of precedence controls the order of merging. Any conflicts or duplicates are automatically resolved.
- Logging makes clear what proejct file and mixins are loaded and merged at startup.
- Like built-in plugins, Ceedling will soon come with built-in mixins available for common build scenarios.
Until this release, Ceedling depended on Rake for most of its command line handling. Rake’s task conventions provide poor command line handling abilities. The core problems with Rake command line handling include:
- Only brief, limited help statements.
- No optional flags to modify a task — verbosity, logging, etc. were their own tasks.
- Complex/limited parameterization (e.g.
verbosity[3]instead of--verbosity normal). - Tasks are order-dependent. So, for example,
test:all verbosity[5]changes verbosity after the tests are run.
Ceedling now offers a full command line interface with rich help, useful order-independent option flags, and more.
The existing new, upgrade, example, and exampples commands remain but have been improved. For those commands that support it, you may now specify the project file to load (see new, related mixins feature discussed elsewhere), log file to write to, exit code handling behavior, and more from the command line.
Try ceedling help and then ceedling help <command> to get started.
Important Notes on the New Command Line:
- The new and improved features for running build tasks — loading a project file, merging mixins, verbosity handling, etc. — are documented within the application command
buildkeyword. A build command line such as the following is now possible:ceedling test:all --verbosity obnoxious --logfile my/path/build.log. Runceedling help buildto learn more and definitely see the next bullet point as well. - The
buildkeyword is assumed by Ceedling. That is, it’s optional.ceedling test:allis the same asceedling build test:all. Thebuildkeyword handling tells the Ceedling application to execute the named build task dynamically generated from your project configuration. - In the transition to remove Rake from Ceedling, two categories of command line interactions now exist. Note this distinction in the
helpheadings.- Application Commands —
help,build,new,upgrade,environment,examples,example,dumpconfig, andversion. These have full help viaceedling help <command>and a variety of useful command line switches that conform to typical command line conventions. - Build & Plugin Tasks — Operations dynamically generated from your project configuration. These have only summary help (listed in
ceedling help) and work just as they previously did. Common command line tasks includingceedling test:allandceedling releaseare in this category.
- Application Commands —
In previous versions of Ceedling, a new, undocumented build directive feature was introduced. Adding a call to the macro TEST_FILE(...) with a C file’s name added that C file to the compilation and linking list for a test executable.
This approach was helpful when relying on a Ceedling convention was problematic. Specifically, #includeing a header file would cause any correspondingly named source file to be added to the build list for a test executable. This convention could cause problems if, for example, the header file defined symbols that complicated test compilation or behavior. Similarly, if a source file did not have a corresponding header file of the same name, sometimes the only option was to #include it directly; this was ugly and problematic in its own way.
The previously undocumented build directive macro TEST_FILE(...) has been renamed to TEST_SOURCE_FILE(...) and is now documented.
Ceedling has been around for a number of years and has had the benefit of many contributors over that time. Preprocessing (e.g. expanding macros in test files and header files to be mocked) is quite tricky to get right but is essential for big, complicated test suites. Over Ceedling’s long life various patches and incremental improvements have evolved in such a way that preprocessing had become quite complicated and often did the wrong thing. Much of this has been fixed and improved in this release. Considerable memory and performance improvements have been made as well.
Previously, if a test executable ran into a segmentation fault (usually caused by memory issues in the code), the entire test executable would report nothing but a simple error. This behavior has been expanded to handle any crash condition and further improved.
By default, a crashed test executable is automatically rerun for each test case individually to narrow down which test case(s) caused the problem. If gdb is properly installed and configured the specific line that caused the crash can be reported.
The :simple and :gdb options for this feature fully and correctly report each test case’s status for a crashed test executable. Crashed test cases are counted as failures. The :none option does not run each test case individually. Instead, in the case of crashed test executable, it marks each test case as a failure reporting that the entire test executable crashed.
See CeedlingPacket) for the new :project ↳ :use_backtrace feature to control how much detail is extracted from a crashed test executable to help you find the cause.
See the documentation for :test_build ↳ :use_assembly to understand how to incorporate assembly code into a given test executable’s build. This complementes Ceedling’s existing ability to incorporate assembly code in a release artifact build.
Ceedling’s previous handling of defaults and configuration processing order certainly worked, but it was not as proper as it could be. To oversimplify, default values were applied in an ordering that caused complications for advanced plugins and advanced users. This has been rectified. Default settings are now processed after all user configurations and plugins.
The Ceedling user guide, CeedlingPacket, has been significantly revised and expanded. We will expand it further in future releases and eventually break it up into multiple documents or migrate it to a full documentation management system.
Many of the plugins have received documentation updates as well.
There’s more to be done, but Ceedling’s documentation is more complete and accurate than it’s ever been.
- Effort has been invested across the project to improve error messages, exception handling, and exit code processing. Noisy backtraces have been relegated to the verbosity level of DEBUG as intended.
- Logical ambiguity and functional bugs within
:pathsand:filesconfiguration handling have been resolved along with updated documentation. - A variety of small improvements and fixes have been made throughout the plugin system and to many plugins.
- The historically unwieldy
verbositycommand line task now comes in two flavors. The original recipe numeric parameterized version (e.g.[4]) exist as is. The new extra crispy recipe includes — funny enough — verbose task namesverbosity:silent,verbosity:errors,verbosity:complain,verbosity:normal,verbosity:obnoxious,verbosity:debug. - This release marks the beginning of the end for Rake as a backbone of Ceedling. Over many years it has become clear that Rake’s design assumptions hamper building the sorts of features Ceedling’s users want, Rake’s command line structure creates a messy user experience for a full application built around it, and Rake’s quirks cause maintenance challenges. Particularly for test suites, much of Ceedling’s (invisible) dependence on Rake has been removed in this release. Much more remains to be done, including replicating some of the abilities Rake offers.
- This is the first ever release of Ceedling with proper release notes. Hello, there! Release notes will be a regular part of future Ceedling updates. If you haven't noticed already, this edition of the notes are detailed and quite lengthy. This is entirely due to how extensive the changes are in the 1.0.0 release. Future releases will have far shorter notes.
- Optional Unicode and emoji decorators have been added for your output stream enjoyment. See the documentation for logging decorators in CeedlingPacket.
- Test suite build order 🔢. Ceedling no longer builds each test executable one at a time. From the tasks you provide at the command line, Ceedling now collects up and batches all preprocessing steps, all mock generation, all test runner generation, all compilation, etc. Previously you would see each of these done for a single test executable and then repeated for the next executable and so on. Now, each build step happens to completion for all specified tests before moving on to the next build step.
- Logging output order 🔢. When multi-threaded builds are enabled, logging output may not be what you expect. Progress statements may be all batched together or interleaved in ways that are misleading. The steps are happening in the correct order. How you are informed of them may be somewhat out of order.
- Files generated multiple times 🔀. Now that each test is essentially a self-contained mini-project, some output may be generated multiple times. For instance, if the same mock is required by multiple tests, it will be generated multiple times. The same holds for compilation of source files into object files. A coming version of Ceedling will concentrate on optimizations to reuse any output that is truly identical across tests.
- Test suite plugin runs 🏃🏻. Because build steps are run to completion across all the tests you specify at the command line (e.g. all the mocks for your tests are generated at one time) you may need to adjust how you depend on build steps.
Together, these changes may cause you to think that Ceedling is running steps out of order or duplicating work. While bugs are always possible, more than likely, the output you see and the build ordering is expected.
- The new internal pipeline that allows builds to be parallelized and configured per-test-executable can mean a fair amount of duplication of steps. A header file may be mocked identically multiple times. The same source file may be compiled identically multiple times. The speed gains due to parallelization help make up for this. Future releases will concentrate on optimizing away duplication of build steps.
- While header file search paths are now customizable per executable, this currently only applies to the search paths the compiler uses. Distinguishing test files or header files of the same name in different directories for test runner and mock generation respectively continues to rely on educated guesses in Ceedling code.
- Any path for a C file specified with
TEST_SOURCE_FILE(...)is in relation to project root — that is, from where you executeceedlingat the command line. If you move source files or change your directory structure, many of yourTEST_SOURCE_FILE(...)calls may need to be updated. A more flexible and dynamic approach to path handling will come in a future update. - Ceedling’s many test preprocessing improvements are not presently able to preserve Unity’s special
TEST_CASE()andTEST_RANGE()features. However, preprocessing of test files is much less frequently needed than preprocessing of mockable header files. Test preprocessing can now be configured to enable only one or the other. As such, these advanced Unity features can still be used in even sophisticated projects.
You may have heard that Ruby is actually only single-threaded or may know of its Global Interpreter Lock (GIL) that prevents parallel execution. To oversimplify a complicated subject, the Ruby implementations most commonly used to run Ceedling afford concurrency and true parallelism speedups but only in certain circumstances. It so happens that these circumstances are precisely the workload that Ceedling manages.
“Mainstream” Ruby implementations — not JRuby, for example — offer the following that Ceedling takes advantage of:
Since version 1.9, Ruby supports native threads and not only green threads. However, native threads are limited by the GIL to executing one at a time regardless of the number of cores in your processor. But, the GIL is “relaxed” for I/O operations.
When a native thread blocks for I/O, Ruby allows the OS scheduler to context switch to a thread ready to execute. This is the original benefit of threads when they were first developed back when CPUs contained a single core and multi-processor systems were rare and special. Ceedling does a fair amount of file and standard stream I/O in its pure Ruby code. Thus, when multiple threads are enabled in the proejct configuration file, execution can speed up for these operations.
Ruby’s process spawning abilities have always mapped directly to OS capabilities. When a processor has multiple cores available, the OS tends to spread multiple child processes across those cores in true parallel execution.
Much of Ceedling’s workload is executing a tool — such as a compiler — in a child process. With multiple threads enabled, each thread can spawn a child process for a build tool used by a build step. These child processes can be spread across multiple cores in true parallel execution.