v2.6

Note!

This release bumps the on-disk minor version of littlefs from lfs2.0 -> lfs2.1.

This change is backwards-compatible, but after the first write with the new version, the image on disk will no longer be mountable by older versions of littlefs.

Read-only operations will not change the on-disk version, and the on-disk version can be explicitly bumped with lfs_fs_mkconsistent if desired.

This is the first on-disk minor version bump for littlefs, and is untested waters, so do open a GitHub issue if this creates any problems.

What's new?

• FCRCs - forward-looking erase-state CRCs (#497)

  Found by @pjsg, littlefs was relying on a bad assumption about the order/atomicity of prog operations to detect incomplete metadata commits. In the case of power-failure, this could lead to littlefs progging the same location multiple times without an erase, leading to best case fixable corruption, worst case significantly reduced data retention.

  The solution implemented here is to store a checksum of the next erased prog-size in every commit. This way any incomplete commits that modify the erased state can be detected.

  In theory this could be implemented in a forward-compatible way, but littlefs currently treats all CRC tags as commit-ending CRC tags, so only updated versions of littlefs will be able to read metadata blocks containing FCRCs. This is the reason for the on-disk minor version bump. On the bright-side, future CRC-related extensions will be forward-compatible.

• A complete overhaul of the testing/bench framework (#752)

  This does not affect users, but involves a long list of needed changes to make it easier to develop/bench littlefs:

  • Tests/benches now run in parallel, and much faster thanks to shared processes.
  • Tests are more reproducible, and narrow bugs can be reproduced locally repeatedly without leaving GDB.
  • More options for power-loss testing. Exhaustive testing of all 1 and 2-deep powerlosses have been added to CI.
  • More scripts related to perf measurements, plotting, tracing, etc.
  • A make help rule to help use/document the Makefile, which is sort of the entry point for littlefs development.
  • Added a CI rule to have @geky-bot comment on each PR with CI measurements (examples here). We'll see if this is an improvement over the GitHub statuses.

  The increased power-loss coverage has already found a couple of difficult to hit bugs (eba5553 and 0b11ce0), and the bench runner will be useful for evaluating littlefs's performance moving forward.

• Added a couple more asserts to help common implementation issues (#801, #802)

• Adopted Brent's algorithm for cycle detection (#809)

  This should significantly speed up cycle detection in the case the metadata linked-list contains a cycle.

• Added lfs_fs_mkconsistent as proposed by @kasper0 (#812)

  lfs_fs_mkconsistent allows users to front-load the internal consistency checks littlefs needs to run before it can write to the filesystem.

  Conveniently this also allows a way to force littlefs to write the new on-disk minor version if that is desired.

• A number of bug fixes, thanks to @ghost, @ajaybhargav, @colin-foster-in-advantage, @rvanschoren, @hgspbs, @sosthene-nitrokey, @kasper0, @Ldd309, and others (#800, #805, #811):

  • #800 - Fixed issues with truncate around block-aligned boundaries
  • #805 - Fixed issue where seeking to end-of-directory return LFS_ERR_INVAL
  • #811 - Fixed issue where lfs_fs_deorphan may run more than needed
  • eba5553 - Fixed issue where half-orphans can hide full orphans in nested power-loss situations
  • 0b11ce0 - Fixed incorrect calculation of extra space needed in mdir blocks
  • ba1c764 - Fixed issue where deorphan could get stuck circling between two half-orpans in power-loss situations

• And a number of links to useful community projects related to littlefs, thanks to @tniessen, @yomimono, @elpiel (#572, #718, #807):

  • littlefs-disk-img-viewer
  • chamelon
  • littlefs2-rust

  If you have a littlefs-related project or a project that uses littlefs, consider opening a PR. Eventually my plan is to move these to a user-contributable wiki.

Changes

	Code	Stack	Structs		Coverage
Default	16556 B	1432 B	772 B	Lines	2289/2469 lines
Readonly	5990 B	448 B	772 B	Branches	1174/1496 branches
Threadsafe	17362 B	1432 B	780 B		Benchmarks
Migrate	18240 B	1736 B	776 B	Readed	29369693876 B
Error-asserts	17192 B	1424 B	772 B	Proged	1482874766 B
				Erased	1568888832 B

259535e Added lfs_fs_mkconsistent
94d9e09 Fixed issue where lfs_fs_deorphan may run more than needed
363a8b5 Tweaked wording of littlefs2-rust link in README.md
e43d381 chore: add littlefs2 crate to README
3e25dfc Added FCRC tags and an explanation of how FCRCs work to SPEC.md
9e28c75 Bumped minor version to v2.6 and on-disk minor version to lfs2.1
4c93600 Added ability to bump on-disk minor version
ca0da3d Added compatibility testing on pull-request to GitHub test action
116332d Added tests for forwards and backwards disk compatibility
f0cc1db Tweaked changeprefix.py to not rename dir component in paths
bf045dd Tweaked link to littlefs-disk-img-viewer to go to github repo
b33a5b3 Fixed issue where seeking to end-of-directory return LFS_ERR_INVAL
384a498 Extend dir seek tests to include seeking to end of directory
b0a4a44 Added explicit assert for minimum block size of 128 bytes
aae897f Added an assert for truthy-preserving bool conversions
e57402c Added ability to revert to inline file in lfs_file_truncate
6dc18c3 Fixed block-boundary truncate issue
d5dc487 Expanded truncate tests to test more corner cases
24795e6 Add missing iterations in tests
7b151e1 Add test scenario for truncating to a block size
ba1c764 Fixed issue where deorphan could get stuck circling between two half-orphans
d1b254d Reverted removal of 1-bit counter threaded through tags
2f26966 Continued implementation of forward-crcs, adopted new test runners
b4091c6 Switched to separate-tag encoding of forward-looking CRCs
91ad673 Cleaned up a few additional commit corner cases
52dd830 Initial implementation of forward-looking erase-state CRCs
1278ec1 Adopted Brent's algorithm for cycle detection
c2147c4 Added --gdb-pl to test.py for breaking on specific powerlosses
801cf27 Tweaked/fixed a number of small runner things after a bit of use
1f37eb5 Adopted --subplot* in plot.py
cfd4e60 Added --subplot* to plotmpl.py
2d2dd8b Added plotmpl.py --github flag to match the website's foreground/background
[b0382fa](https://github.com/littlefs-pro...

v2.5.1

Configuration	Code	Stack	Structs	Coverage
Default	15722 B (+0.0%)	1312 B (+0.0%)	772 B (+0.0%)	92.6% of 2384 lines (-0.1%)
Readonly	5746 B (+0.1%)	352 B (+0.0%)	772 B (+0.0%)
Threadsafe	16516 B (+0.0%)	1312 B (+0.0%)	780 B (+0.0%)
Migrate	17414 B (+0.0%)	1648 B (+0.0%)	776 B (+0.0%)
Error-asserts	16326 B (+0.0%)	1304 B (+0.0%)	772 B (+0.0%)

740d9ac Fix readme Mbed link
d08f949 Fixed lfs_dir_fetchmatch not propogating bd errors correctly in one case
9e965a8 lfs_util: Fix endiannes conversion when LFS_NO_INTRINSICS is set
eb9f4d5 Fix invalid block size reporting.
a25681b Improve lfs_file_close usage description
47914b9 Fixed self-assign warnings discovered by clang
30175de Remove -Wshadow -Wjump-misses-init -Wundef
2374762 Added clang build step to CI
a405c32 lfs_filebd_sync: fix compilation on Windows
9af63b3 Fix buffer overflow in tests when using a large block size
6c720dc Fix unused function warning with LFS_NO_MALLOC

v2.5

What's new?

• Removed all recursion (#658)

  On paper, littlefs promises O(1) RAM consumption, however in practice, this hasn't quite been true due to the use of recursion in several parts of the implementation. 2 instances of recursion were bounded at runtime, but 1 instance could be unbounded, though this would require many block relocations to all occur simultaneously.

  These pieces have now been rewritten to strictly avoid recursion, at the cost of some code size (<=5.1%). This means littlefs now lives up to its promise of bounded RAM, and the total RAM usage can be easily evaluated using static tooling.

• Added several new scripts for measuring RAM usage, including detection of future recursion (#658)

  These have also been integrated into CI, so data, stack, and struct usage will now be reported with all CI jobs.

• Several corner-case performance improvements thanks to @invoxiaamo and @robekras:

  • lfs_dir_traverse no longer exhibits a nasty O(n^3) performance spike when a rename occurs at the same time a directory is being compacted (#666)

    @invoxiaamo notes a reduction in lfs_dir_traverse calls by 97.8% (3248 vs 148393), with a number of other proposal to improve this with additional RAM caching in #621.

    This change also reduces stack consumption of lfs_dir_traverse by ~25%.

  • lfs_file_seek now avoids flushing when seeking inside the cache while reading (#632)

    @robekras notes a ~90% (2 sec vs 20 sec) speedup when loading a sparse binary font file into LVGL.

• Several improvements to API robustness thanks to @colin-foster-in-advantage, @nnayo, @Johnxjj, and @lmapii:

  • Attempting to mount a littlefs image with a block_size or block_count that does not match the configuration will now error (#584)

  • lfs_file_open is now omitted when building with LFS_NO_MALLOC, previously this would always be a runtime error (#614)

  • Seeking to a negative offset is now an error, instead of underflowing (#630)

  • lfs_file_open now returns LFS_ERR_NAMETOOLONG if it can't fit the name + file metadata into a metadata block, previously this would return LFS_ERR_NOSPC (#638)

• Thanks to @m8ddin, lfs_filebd now works on Windows (#643)

• The copyright notice has been appended to reflect ownership changing to "the littlefs authors" (#657)

Changes

Configuration	Code	Stack	Structs	Coverage
Default	15722 B (+12.7%)	1312 B (-∞%)	772 B (+0.0%)	92.7% of 2388 lines (-2.0%)
Readonly	5742 B (+7.4%)	352 B (+4.7%)	772 B (+0.0%)
Threadsafe	16516 B (+10.9%)	1312 B (-∞%)	780 B (+0.0%)
Migrate	17414 B (+11.5%)	1648 B (-∞%)	776 B (+0.0%)
Error-asserts	16326 B (+12.3%)	1304 B (-∞%)	772 B (+0.0%)

148e312 Bumped minor version to v2.5
abbfe8e Reduced lfs_dir_traverse's explicit stack to 3 frames
1e038c8 Fixes to use lfs_filebd on windows platforms
c2fa1bb Optimization of the rename case.
3b62ec1 Updated error handling for NOSPC
b898977 Set the limit, the cursor cannot be set to a negative number
cf274e6 Squash of CR changes
425dc81 Modified robekras's optimization to avoid flush for all seeks in cache
a6f01b7 Update lfs.c
2db5dc8 Update copyright notice
8109f28 Removed recursion from lfs_dir_traverse
fedf646 Removed recursion in file read/writes
84da4c0 Removed recursion from commit/relocate code path
554e4b1 Fixed Popen deadlock issue in test.py
fe8f3d4 Changed./scripts/struct.py to organize by header file
316b019 In CI, determine loop devices dynamically to avoid conflicts with Ubuntu snaps
8475c80 Limit ./scripts/structs.py to report structs in local .h files
563af5f Cleaned up make clean
3b495ba Fixed spurious CI failure caused by multiple writers to .o files
e4adefd Fixed spurious encoding error
9d54603 Added new scripts to CI results
7ea2b51 A few more tweaks to scripts
55b3c53 Added ./script/summary.py
eb8be9f Some improvements to size scripts
50ad2ad Added make *-diff rules, quick commands to compare sizes
0a2ff3b Added scripts/structs.py for getting sizes of structs
d7582ef Changed script's CSV formats to allow for merging different measurements
f4c7af7 Added scripts/stack.py for viewing stack usage
20c58dc Added coverage-sort to scripts/coverage.py
f5286ab Added scripts/calls.py for viewing the callgraph directly
2cdabe8 Split out scripts/code.py into scripts/code.py and scripts/data.py
b045436 Added size-sort options to scripts/code.py
e334983 don't use lfs_file_open() when LFS_NO_MALLOC is set
487df12 Fail when block_size doesn't match config
3efb8e4 Fail mount when the block size changes

v2.4.2

Code size	Code size (readonly)	Code size (threadsafe)	Code size (migrate)	Code size (error-asserts)	Coverage
14592 B (-0.1%)	5634 B (+0.9%)	15414 B (-0.1%)	16288 B (-0.0%)	15204 B	94.7% of 1758 lines (+0.0%)

9c7e232 Fixed missing definition of lfs_cache_drop in readonly mode
03f088b Tweaked lfs_file_flush to still flush caches when build under LFS_READONLY
e955b9f Fix lfs_file_seek doesn't update cache properties correctly in readonly mode. Invalidate cache to fix it.
1363c9f fix bug:lfs_alloc will alloc one block repeatedly in multiple split
5bc682a Typo
1877c40 Indent sub-portions of tag fields
e29e7ae Specify unit of the size members of the lfs_config struct
4977fa0 Fix spelling errors
fdda3b4 Always zero rambd buffer before first use
fb2c311 Fix compiler warnings when LFS_READONLY defined

v2.4.1

Code size	Code size (readonly)	Code size (threadsafe)	Code size (migrate)	Code size (error-asserts)	Coverage
14600 B (+0.0%)	5586 B (+0.0%)	15422 B (+0.0%)	16296 B (+0.0%)	15212 B	94.7% of 1758 lines (+0.0%)

1e423ba Fix link to test block device
3bee4d9 scripts/test.py: Fix infinite busy loops on macOS

v2.4

What's new?

• Graceful LFS_ASSERT (#514)

  Thanks to @mon, @davidefer, and everyone else who provided feedback, littlefs now supports overriding LFS_ASSERT with a macro that can return an error instead of halting.

  This doesn't change the behavior LFS_ASSERT itself, but just means littlefs only uses LFS_ASSERT in functions that return a signed error code. This is an internal rule littlefs will follow, and thanks to @mon we now have a CI job to prevent breaking this behavior. Outside of littlefs, you're free to use LFS_ASSERT in non-erroring functions, you'll just lose the ability to override LFS_ASSERT in this way.

• New metadata_max config (#502)

  A great idea from @mon, metadata_max provides a workaround for performance issues littlefs faces with large block sizes (#203).

  metadata_max reduces the performance hit of metadata compaction by artificially limiting how much of a metadata block we use. This improves performance since we there is less metadata to compact, but does mean we waste whatever space in the block we don't use. It's interesting tradeoff since the devices where block sizes are large enough for this to be an issue (NAND) also have enough storage to throw at this problem. It's not the best solution, but may make it possible to get past some of these performance bottlenecks.

  @mon notes a 10x speedup in directory traversal/file deletion and a 2x-4x speed up in file reads on a 128KiB NAND block device (MT29F) with metadata_max set to 4KiB.

• Adopt GitHub Actions for CI, and a number of script/Makefile improvements (#516)

  Since this is primarily for development and doesn't affect most users, I will just point to the PR notes in #516.

  If everything works, you will probably already see the new code size/coverage measurements automatically appended to these release notes.

  One of the more interesting things to note is the addition of a post-release process to automatically create PRs against dependent repos. This should help keep dependents such as littlefs-fuse no longer constantly-out-of-date. If you want to add a repo to receive automatic PRs feel free to create a PR against the littlefs.post-release repo.

• Thanks to @embeddedt, lfs_file_seek has a new optimization to avoid flushing caches when seeking to the same position (#511)

• littlefs now performs a sanity check in lfs_format after compacting the superblock, instead of before (#508)

Changes

Code size	Code size (readonly)	Code size (threadsafe)	Code size (migrate)	Code size (error-asserts)	Coverage
14600 B	5586 B	15422 B	16296 B	15212 B	94.7% of 1758 lines

3d4e4f2 Bumped minor version to v2.4
c0cc0a4 Enabled overriding of LFS_ASSERT/TRACE/DEBUG/etc
c9eed1f Add test to ensure asserts can return
e7e4b35 lfs_fs_preporphans ret int for graceful LFS_ASSERT
21488d9 Fixed incorrect documentation in test.py
10a0883 Moved lfs_mdir_isopen behind LFS_NO_ASSERT
47d6b2f Removed unnecessary truncate condition thanks to new seek optimization
745d98c Fixed lfs_file_truncate issue where internal state may not be flushed
3216b07 Use lfs_file_rawsize to calculate LFS_SEEK_END position
6592719 Removed .travis.yml
c911061 Added post-release script, cleaned up workflows
104d651 Reduced build sources to just the core littlefs
6d3e4ac Brought over the release workflow
9d65460 Fixed a recompilation issue in CI, tweaked coverage.py a bit more
b84fb6b Added BUILDDIR, a bit of script reworking
887f366 Switched to lcov for coverage collection, greatly simplified coverage.py
eeeceb9 Added coverage.py, and optional coverage info to test.py
b2235e9 Added GitHub workflows to run tests
6bb4043 Skip flushing file if lfs_file_rawseek() doesn't change position
2b80453 Moved sanity check in lfs_format after compaction
d804c2d Added scripts/code_size.py, for more in-depth code-size reporting
37f4de2 Remove inline_files_max and lfs_t entry for metadata_max
6b16daf Add metadata_max and inline_file_max to config

v2.3

What's new?

Thanks to @maximevince, @renesas-billgesner, and @roykuper13, we now have two impactful compile-time options for littlefs!

• LFS_READONLY

  Define LFS_READONLY to enable read-only mode. This will limit littlefs to functions that do not modify the disk, but will reduce the code cost to ~1/3. This is a great addition for bootloaders or other applications where a small, read-only storage blob is useful.

• LFS_THREADSAFE

  Define LFS_THREADSAFE to enable thread-safe mode. Note this needs the lock and unlock callbacks in the lfs_config struct.

  When defined, littlefs will call the related lock and unlock callbacks when necessary to synchronize the filesystem. Allowing easy integration into an RTOS without needing an OS or wrapper layer.

In addition, this release brings in a number of improvements:

• More strict user asserts on common file/dir open mistakes thanks to @NoahGorny (#443)
• More efficient, 8-byte lfs_bd_cmp thanks to @gmpy (see #395 for measurements)
• A number of wear-leveling fixes thanks to @guiserle and @gtaska (#487)
• Fixes for allocation-eviction math thanks to @tim-nordell-nimbelink (#490)

Here are the current code costs for the different littlefs configurations in CI:

	Arm Thumb	x86-64	MIPS	PowerPC
default	14.53 KB	25.55 KB	28.76 KB	23.53 KB
LFS_READONLY	5.56 KB (-61.73%)	9.53 KB (-62.72%)	11.08 KB (-61.47%)	8.90 KB (-62.17%)
LFS_THREADSAFE	15.36 KB (+5.75%)	27.79 KB (+8.76%)	30.98 KB (+7.73%)	25.64 KB (+8.99%)

Changes

6a70127 Renamed internal lfs_*raw -> lfs_raw* functions
288a5cb Bumped minor version to v2.3
2bb5234 Moved lfs_mlist_isopen checks into the API wrappers
7388b29 Deprecate LFS_F_OPENED and use lfs_mlist_isused instead
a99a93f Added thread-safe build+size reporting to CI
45afded Moved LFS_TRACE calls to API wrapper functions
00a9ba7 Tweaked thread-safe implementation
fc6988c make raw functions static. formatting tweaks
d0f055d Squash of thread-safe PR cleanup
2efebf8 Added read-only build+size reporting to CI
754b4c3 Squash of LFS_READONLY cleanup
008ebc3 Add lfs_mlist_append/remove helper
b8dcf10 Changed lfs_dir_alloc to maximize block cycles for new metadata pairs
0aba71d Fixed single unchecked bit during commit verification
0ea2871 Fixed typo in scripts/readtree.py
d04c139 Fixed allocation-eviction issue when erase state is multiple of block_cycles+1
f215027 Switched to CRC as seed collection function instead of xor
1ae4b36 Removed unnecessary randomization of offsets in lfs_alloc_reset
480cdd9 Fixed incorrect modulus in lfs_alloc_reset
6303558 Use LFS_O_RDWR instead of magic number in lfs_file_* asserts
4bd653d Assert that file/dir struct is not reused in lfs_file_opencfg/lfs_dir_open
8e6826c Add LFS_READYONLY define, to allow smaller builds providing read-only mode
10ac6b9 add thread safe wrappers
87a2cb0 Fix assert
6d0ec5e Added littlefs-python to the related projects section
64f70f5 lfs_bd_cmp() compares more bytes at one time

v2.2.1

5a9f38d Remove -Wno-missing-field-initializers
1b033e9 Fix -Wmissing-field-initializers

v2.2

What's new?

This release brings in a revamped testing framework intended to improve the rate of bug isolation and fixes. There are new runtime contexts for tests such as simulated power-loss and Valgrind based memory checking, and additional scripts for printing out metadata to debug the filesystem (readtree.py/readmdir.py/readblock.py in the scripts directory). This also comes with a large number of bug fixes as detailed in #372.

More details on this can be found in the overly wordy #372.

Other changes:

• Introduced rambd and filebd, two block devices that replace emubd for simulating block devices locally
• Changed asserts on corrupted filesystem to return errors, allowing additional user recover
• A number of quality-of-life improvements to logging statements
• Reenabled cache bypassing if the filesystem is handling data that is aligned to the block device's requirements
• Did I mention a large number of bug fixes? There's a lot of bugs fixed in this release, see #372

Changes

6372f51 Allow debug prints without VA_ARGS
6622f3d Bumped minor version to v2.2
5137e4b Last minute tweaks to debug scripts
ff84902 Moved out block device tracing into separate define
f9dbec3 Added test case catching issues with errors during a lookahead scan
f17d3d7 Minor cleanup
5e5b5d8 (chore) updates from PR, we decided not to move forward with changing v1 code since it can be risky. Let's improve the future! Also renamed and moved around a the lookahead free / reset function
d498b9f (bugfix) adding line function to clear out all the global 'free' information so that we can reset it after a failed traversal
4677421 Added "evil" tests and detecion/recovery from bad pointers and infinite loops
cb26157 Change assert to runtime check.
a7dfae4 Minor tweaks to debugging scripts, fixed explode_asserts.py off-by-1
50fe8ae Renamed test_format -> test_superblocks, tweaked superblock tests
0990296 Limited byte-level tests to native testing due to time
d04b077 Fixed minor things to get CI passing again
c7987a3 Restructured .travis.yml to span more jobs
dcae185 Fixed typo in LFS_MKTAG_IF_ELSE
f4b17b3 Added test.py support for tmpfs-backed disks
9f546f1 Updated .travis.yml and added additional geometry constraints
b69cf89 Fixed CRC check when prog_size causes multiple CRCs per commit
02c84ac Cleaned up dependent fixes on branch
6530cb3 Fixed lfs_fs_size doubling metadata-pairs
fe957de Fixed broken wear-leveling when block_cycles = 2n-1
6a55084 Modified readmdir/readtree to make reading non-truncated data easier
f9c2fd9 Removed file outlining on ENOSPC in lfs_file_sync
44d7112 Fixed tests/.toml. in .gitignore
77e3078 Added/fixed tests for noop writes (where bd error can't be trusted)
517d341 Fixed more bugs, mostly related to ENOSPC on different geometries
4fb1883 Update SPEC.md
c8e9a64 Indicate C99 standard as target for LittleFS code
aab6aa0 Cleaned up test script and directory naming
52ef0c1 Fixed a crazy consistency issue in test.py
b9d0695 Rewrote explode_asserts.py to be more efficient
a5d614f Added tests for power-cycled-relocations and fixed the bugs that fell out
f4b6a6b Fixed issues with neighbor updates during moves
9453ebd Added/improved disk-reading debug scripts
fb65057 Restructured block devices again for better test exploitation
ecc2857 Migrated bad-block tests
5181ce6 Migrated the first of the tests with internal knowledge
b06ce54 Migrated the bulk of the feature-specific tests
1d2688a Migrated test_files, test_dirs, test_format suites to new framework
eeaf536 Replaced emubd with rambd and filebd
626006a Fix incorrect comment on lfs_npw2
53d2b02 Added reentrant and gdb testing mechanisms to test framework
ed8341e Reworked permutation generation in test framework and cleanup
f42e007 Created initial implementation of revamped test.py
5a12c44 Revert "Don't bypass cache in lfs_cache_prog() and lfs_cache_read()"

v2.1.4

ce2c01f Fixed lfs_dir_fetchmatch not understanding overwritten tags
0197b18 Fixed issue with superblock breaking lfs_dir_seek
d6dc728 Fixed some issues in lfs_migrate
aeff2a2 Stop wear-leveling during migration
aae22c8 Fixed issue with directories falling out of date after block relocation
60e67ae Fixed implicit change-of-sign warning in lfs_dir_fetch
64dedee prepare upstream bugfix of lfs
5925db4 Fix some LFS_TRACE format specifiers
ab56dc5 README: fix incorrect description