vfs: file system abstraction

[jnohlgard]

This PR provides a basic virtual file system (VFS) layer for RIOT. The intention is to provide a much simpler layer than what was proposed in #2474 and #1265.

The overall design goals are:

• Provide implementations for all newlib "file" syscalls
• Keep it simple, do not add every possible file operation from Linux VFS.
• Easy to map existing file system implementations for resource constrained systems onto the VFS layer API
• Avoid keeping a central enum of all file system drivers that has to be kept up to date with external packages etc.
• Use POSIX <errno.h> numbers as negative return codes for errors, avoid the global errno variable.
• Only absolute paths to files (no per-process working directory)
• No dynamic memory allocation

The file systems for resource constrained systems that I have been looking at (so far) are Contiki CFS and SPIFFS.
I have also been looking at Linux kernel headers and books/articles on writing Linux device drivers for more ideas for the design.

The API should be easy to understand for users who are familiar with the POSIX file functions (open, close, read, write, fstat, lseek).

The VFS layer keeps track of mounted file systems and open files, the vfs_open function searches the array of mounted file systems and dispatches the call to the file system instance with the longest matching mount point prefix.
Subsequent calls to vfs_read, vfs_write, etc will do a look up in the table of open files and dispatch the call to the correct file system driver for handling.

vfs_mount takes a string containing the mount point, a file system driver specification (struct file_system), and an opaque pointer that only the FS driver knows how to use, which can be used to keep driver parameters in order to allow dynamic handling of multiple devices.

Also included is a simple file system implementation of a static const file system called ConstFS. This was created as a tool to assist in unit testing of the VFS layer, and as an exercise in implementing file system drivers, and to provide an example for developers who wish to write their own file system driver to hook into the VFS layer.

TODO:

• STDIN, STDOUT, STDERR through vfs_read/write
• Write the newlib syscall wrappers
• Elaborate the design thoughts in the Doxygen documentation
• Explain clearly how to use the private_data members when implementing a file system driver
• auto_init stuff?
• Shell commands (mount will not be supported due to the dynamic nature of the vfs_mount function)
  • read file
  • write file
  • list mounted file systems and space utilization (vfs df)
  • list files in a directory
• Improve constness to allow for keeping large structures like vfs_file_ops_t in flash.
• ConstFS: Implement fstat, lseek, update read to use the file offset
• Mutex for the open files table
• fcntl - change flags on already open FDs
• ioctl - communicate with special devices - not needed since kernel is rebuilt every time, easy to use device driver directly
• stat (may wrap open+fstat+close sequence as a fall back)
• statvfs, fstatvfs - information about a mounted file system
• More unit tests
  • vfs_open
  • vfs_close
  • vfs_read
  • vfs_write
  • vfs_lseek
  • vfs_fstat
  • vfs_fcntl
  • vfs_mount
  • vfs_umount
  • vfs_rename
  • vfs_unlink
  • vfs_mkdir
  • vfs_rmdir
  • vfs_bind
  • vfs_opendir
  • vfs_closedir
  • vfs_readdir
  • vfs_normalize_path
  • vfs_iterate_mounts
  • vfs_stat
  • vfs_statvfs
  • vfs_fstatvfs
  • newlib syscall wrappers
• DevFS device file system
  • basic functionality
  • auto_init for mounting
• DevFS support in drivers
  • /dev/uartX UART devices
  • /dev/nvramX NVRAM devices - basic implementation done
  • /dev/flashX Flash devices (requires: Flash API, see mtd flash: add a generic low level flash interface [RFC] #5624, Flash interface #2239)
  • /dev/random periph/random
• import xxd-like dumper from other branch as shell command
• Directory support
  • opendir
  • readdir
  • closedir
  • unit tests for directories
• vfs_rename

Not implemented:

• Simultaneous access to the same file.

How to try it

Right now the only place to run the code is through a newly added embunit unit test batch:

make -C tests/unittests tests-vfs term BOARD=native

(also successfully tested on BOARD=mulle)

Open questions

• Does it make sense to add an "inode" layer to separate the file name handling from the file opening?
• Should mode be given as mode_t or int? (newlib uses int, POSIX uses mode_t which is unsigned long on 32 bit platforms)
• Should mounted file systems be changed into a linked list? (originally was statically allocated array with fixed maximum mount point length) - implemented

[kaspar030]

Nice! Thank you for tackling this. Looks very nice already!

Some thoughts that came up while skimming the diff:

• did you consider extending structs by embedding instead of using pointers, (like netdev2 does)?
• any reason you're using "struct xxx..." within structs, instead of the typedefs?

[jnohlgard]

@kaspar030

• did you consider extending structs by embedding instead of using pointers, (like netdev2 does)?

My thoughts were that some of these structs should be constant and static, e.g. the file system driver, and should not consume valuable RAM, while other parts are variable. Right now I think all of it is kept in RAM, but the goal is to keep the static parts in flash instead and using pointers from the RAM parts to the flash parts.
-- updated: file operations and file system operations are const and kept in flash

• any reason you're using "struct xxx..." within structs, instead of the typedefs?

No particular reason, I started using them in one place to get a pointer to the same type of struct, and then it just kind of kept on. I'll see if it still compiles if I modify it to use the typedefs instead.
-- updated: modified to use structname_t instead of struct structname

[Kijewski]

break

[jnohlgard]

addressed

[jnohlgard]

updated todo-list with some more ideas.

[vincent-d]

This looks very nice.

I was working on the same topic, but my work is much less advanced than yours. I'm trying to get SPIFFS working and we worked on a low level flash interface as well. I'll push some work soon and will try to rebase my SPIFFS port on top of your vfs.

Thanks for that work

[jnohlgard]

@vincent-d Nice! I have had plans to add SPIFFS but I did not feel that I knew where to begin, so I started with this instead, it's good to see that others are working towards the same goal.
I am currently under way to add a device file system, DevFS, similar to /dev on Linux, in order to be able to access low level devices from the shell, such as NVRAM. Maybe we can fit the generic flash driver onto that as well? (see TODO in top post)

[jnohlgard]

squashed, WIP commit list was getting out of hand.

[kaspar030]

How do we deal with directories?

[jnohlgard]

Updated with more work on DevFS, including:

• DevFS auto_init mounts /dev
• Shell command vfs to access the VFS mounted file systems from the shell
• Newlib fixes
• basic fcntl implementation
• VFS file operations implementation for /drivers/nvram to access the contents of NVRAM devices via VFS.
• Configured NVRAM VFS on DevFS for Mulle (/dev/mulle-fram).
• improved unit tests

[jnohlgard]

@kaspar030 not sure. CFS and SPIFFS both implement specific opendir, closedir, readdir. Maybe those functions should be added as part of vfs_file_system_ops_t, what is your opinion?

[vincent-d]

This should not be done here but only in vfs_open, otherwise the unlink also increments the counter and umount fails afterwards.

[jnohlgard]

Actually, this is on purpose as a kind of semaphore to avoid races between umount and open/unlink. There were a few missing decrements of open_files in vfs_unlink though that I have fixed now.

[jnohlgard]

I'll do an update tomorrow morning with opendir, readdir, closedir support for devfs and constfs (and the VFS support code of course)

[jnohlgard]

Updated with the directory interface. No unit tests yet or ConstFS, DevFS support for directories.

[miri64]

I know FAT is not the most ideal FS for our purposes with all its layers of backwards compatibility, but we also should consider porting the FatFS library as a package using this interface. After all: FAT is still the file system type most SD cards are shipped with. I saw this library used on systems that are way more constraint than the ones RIOT targets (the CCC's r0ket badge e.g.), so space shouldn't be an issue.

[miri64]

Does this need a re-ACK?

[jnohlgard]

@miri64 Needs at least someone to approve the workaround for the mips board in the beginning of fcntl.h and statvfs.h

[jnohlgard]

@vincent-d Does your ACK hold?

[vincent-d]

Yes! ACK

[miri64]

qemu error on Jarvis seems unrelated and Murdock (1) is still the imperative CI, so merge at will.

[jnohlgard]

Yay! :D

[miri64]

Congratz \o/

[vincent-d]

Great!!

[kaspar030]

Congrats and a lot of thanks to @gebart! Good job!

[emmanuelsearch]

great!!

[kYc0o]

Super! Thanks @gebart !