[HELP] FFI compatiblity issues

[no1wudi]

Description

Backgound

I'm porting Rust libstd to NuttX, and I've run into a problem with FFI.

Please check the early work in progress: apache/nuttx-apps#2487

For Rust, the libstd is split into two parts: libcore and libstd.
The libcore is a minimal runtime that can be used on bare metal systems,
and it doesn't depend on any OS-specific functionality. The libstd,
on the other hand, depends on the OS-specific functionality, such as
threading, networking, etc.

libstd uses libc for FFI, and them usually shipped as static libraray.
And I've run into a problem with that.

Problem

NuttX is highly configurable, the good part of it is that you can disable
anything you don't need, but the bad part is that you can't assume anything
about the system. For example, the size of complex types like struct timespec
can be different:

nuttx/include/time.h

Lines 113 to 117 in f7adb52

	struct timespec
	{
	time_t tv_sec; /* Seconds */
	long tv_nsec; /* Nanoseconds */
	};

The time_t type can be 32 or 64 bits, and the struct timespec can be
8 or 16 bytes. This is a problem for Rust, because it's hard to catch this at compile time.

And for libc, there are many complex types will be used in both Rust side and C side,
for current implmentation of libc crate, they defiend all of these types again with #[repr(C)],
which means that they have the same layout as in C:

https://github.com/no1wudi/libc/blob/b395ec66982ae160dba1d7061c51311f32052633/src/unix/nuttx/mod.rs#L32-L56

Maybe for other OS, the data definitons are stable enough, but for NuttX, it's not.

For example, these data types are used in libc crate:

nuttx/include/pthread.h

Lines 221 to 243 in f7adb52

	struct pthread_attr_s
	{
	uint8_t priority; /* Priority of the pthread */
	uint8_t policy; /* Pthread scheduler policy */
	uint8_t inheritsched; /* Inherit parent priority/policy? */
	uint8_t detachstate; /* Initialize to the detach state */
	#ifdef CONFIG_SCHED_SPORADIC
	uint8_t low_priority; /* Low scheduling priority */
	uint8_t max_repl; /* Maximum pending replenishments */
	#endif
	#ifdef CONFIG_SMP
	cpu_set_t affinity; /* Set of permitted CPUs for the thread */
	#endif
	FAR void *stackaddr; /* Address of memory to be used as stack */
	size_t stacksize; /* Size of the stack allocated for the pthread */
	#ifdef CONFIG_SCHED_SPORADIC
	struct timespec repl_period; /* Replenishment period */
	struct timespec budget; /* Initial budget */
	#endif
	};

It's hard to map CONFIG_SCHED_SPORADIC and CONFIG_SMP and mores one by one in Rust side, espcially in a cross-compiling environment.

And full list of data types used in libc crate can be found here:
https://github.com/no1wudi/libc/blob/libc-0.2/src/unix/nuttx/mod.rs

Possible Solutions

1. Reserving space in Rust side for all possible data types

This is the most simple solution, but it's not very elegant. Many strucutres can be used as opaue data blobs in Rust side,
they don't need to be defined excatly same as in C side, just keepping the sizse of them is enough, for example:

https://github.com/no1wudi/libc/blob/b395ec66982ae160dba1d7061c51311f32052633/src/unix/nuttx/mod.rs#L79-L81

The drawback of this solution:

1. Waste memory space, especially for embedded systems.
2. Hard to maintain, if the data type is changed in C side, and exceeeding the reserved size, it will cause undefined behavior.

2. Mapping all possible conigurations in Rust side

The drawback of this solution:

1. It's hard to maintain the configs in both Rust and NuttX side, or other lanuages in the future.
2. The libstd and libcore must be build from source code each time.

3. Isolate the exported data types from NuttX

Do not use the libc data typs directly in NuttX intenral if any funtionality is conigurable, instead, define a new data type for it.

The drwa back of this solution:

1. It's a larage work to change all the exported data types from NuttX.

4. Let Rust runtime only avaliable with specific configurations

For example, add a special config like CONFIG_RUST_RUNTIME in NuttX, and let it depenends on all the configurations that Rust runtime needs and match the data types with libc crate:

config RUST_RUNTIME
    bool "Enable Rust Runtime"
    depends on !ARCH_FPU
    depends on SYSTEM_TIME64
    depends on ANYOTHER_ESSENTIAL_CONFIGS

Discussion

I prefer the solution 4, but it's less flexible if you do not want to use full feaure of Rust libstd.

Any sugestions are welcome!

Verification

• I have verified before submitting the report.

[xiaoxiang781216]

@lupyuen could you look at this issue?

[lupyuen]

Hi @no1wudi thanks for the interesting question! Sorry I don't strong opinions about this, but here's my gut feel:

Solution 1: Reserving space in Rust side for all possible data types

This approach says that Rust will reserve sufficient Struct Space to fit the largest possible Struct Size (of that specific struct, across all platforms). (Like this)

I agree with you: Every byte is precious for Embedded Apps, so I don't think NuttX Community will accept this.

Solution 2: Mapping all possible configurations in Rust side

This approach says that every possible Data Type in C, across all platforms, will be mapped to the equivalent Data Type in Rust. (Sounds like bindgen, see below)

I agree with you, this is hard to maintain. And it introduces extra overheads into the build process.

Solution 3: Isolate the exported data types from NuttX

This sounds like we're creating Wrapper Types, hiding a C Data Type inside a Rust Data Type. This will be difficult to maintain, and might impact the runtime performance.

Solution 4: Rust runtime is only supported for specific configurations

This approach says that Rust will support only specific NuttX Configurations (e.g. !ARCH_FPU && SYSTEM_TIME64). Rust Data Types will be matched to only those specific C Data Types that we support. (Hope I understand this correctly)

As mentioned, it's less flexible and we might not allow the full functionality of Rust Standard Library. But it seems adequate for now, and I agree with this approach.

I'm curious about this potential solution...

Solution 5: Use bindgen to generate Rust Types at Compile Time

We could run bindgen on all the NuttX Header Files (including specific configs like !ARCH_FPU && SYSTEM_TIME64). This will produce the equivalent Rust Data Types for every C Data Type. But the build will get complicated. And I'm not sure if a Rust Standard Library will allow bindgen internally?

I think we can proceed with Solution 4, and maybe later evolve to Solution 5. Hopefully better tools will emerge.

Here's an example: LVGL Graphics Library has a similar problem of supporting multiple embedded platforms with different type sizes. They solve it by running bindgen to generate Custom Rust Bindings for each embedded platform: https://github.com/lvgl/lv_binding_rust

Further Research

This problem might be common to any Embedded OS with POSIX? Perhaps we could do some research and understand how they handle it:

• Rustix Wrapper for POSIX
• Zephyr OS
• Rust for Linux Kernel (does it use bindgen?)
• (Maybe) ESP32 with Rust Standard Library

Hi @acassis wonder if you have anything to add?

[acassis]

@lupyuen I also don't have strong opinion too! I agree with you Solution 4 could be an initial solution until we get Solution 5 working.

[no1wudi]

@lupyuen @acassis Thanks for your feedback!

In my opinion, bindgen is very suitable for further package since libstd and libc only provide the standard library, for the OS specific function, such as SPI/IIC or many other device driver we can use these approache to create a crate that provide the NuttX spcific functions.

For the libstd itselfs, it don't need to interact with NuttX directly, but libc need, libstd built on top of the libc crate, if we use bindgen solution for libc, the entire libstd and libc need to be build from source with a configured NuttX source tree by a unstable option -Zbuild-std.

This means user must use a nighlty rust toolchain, which may be a problem for production environment.

So maybe we should combine Solution 1 and Solution 4 to create a relatively stable binary interface, that allow the libstd and libc without NuttX enviroment, follow the Rust way to deliver the libstd with prebuild library?

I can provide a patch to demonstrate the specific approach in a few days.

[lupyuen]

@no1wudi Thanks that sounds good!

if we use bindgen solution for libc, the entire libstd and libc need to be build from source with a configured NuttX source tree by a unstable option -Zbuild-std

Sorry I'm curious: Today Rust Apps won't compile for QEMU RISC-V 32-bit because it needs a Custom Target for rv32gc: https://lupyuen.github.io/articles/rust4

Will this still happen when we have implemented libstd for NuttX? I'm just wondering if we eventually need to recompile libstd anyway, since the NuttX Targets so different from the typical ones.

[no1wudi]

@lupyuen I can add it to the Rust side as rust-lang/rust#127755 does in next patch, so all these target in the support list will available out of box.

[lupyuen]

@no1wudi That's great thanks!

[no1wudi]

@lupyuen #13245 shows the structure and data type handling, can you take a look about it ?

[no1wudi]

rust-lang/libc#3920

I'm not very familiar with esp-idf's Rust implemetation now, but according to the error log, it's very close to #13245 for compatility checking.

[no1wudi]

rust-lang/rust#130595

Now I get the Rust app with std library support on my local machine, the size of the full library is around 240K, for a hello world example with println! is about 70K.

It's comparable to libcxx.

[cederom]

• Thanks @no1wudi !! :-)
• For me Solution 4 seems most reasonable, compatible, least invasive, easiest to implement and maintain - enabling Rust will also enable all required components necessary :-)
• Solution 5 proposed by @lupyuen seems most versatile in the long term. When Solution 4 is ready then Solution 5 may be considered if it brings additional benefits.
• Regarding the Rust Toolchain there are long discussions that I follow on the FreeBSD mailing lists. Linux in-kernel maintainer resigned. And there are strong oppositions (including myself) to include Rust into FreeBSD kernel / base at this point. The problem is unstable toolchain (moving target -> dependency nightmare) that needs to be built before building the OS and I guess we cannot skip it here in NuttX too. There must be a way to compile everything from the sources and do not depend on external binary toolchains (but binary toolchains are the option that saves time I know). If someone chooses to work with the Rust they must also follow the disadvantages of the solution (i.e. build the toolchain). Rust is not a solution that has advantages only as advertised (not yet maybe time will tell).
• Regarding the Rust on RISC-V there are problems on FreeBSD too [1]. That comes from required ABI that has been set for FreeBSD 11 in the RISC-V Rust upstream and everyone now needs to hand build in the COMPAT_FREEBSD11 (current - 4) which is not default. I work on FreeBSD 13 on my desktop, 14 is stable now (I have it on my laptop), 15 is the current / master branch, so any ABI that will be selected will cause pain. And the funny thing is one package librsvg-rust is required to build gtk3 and that propagates to half of the packages dependencies :D :D :D
• I would be really cautious with this Rust stuff. Seems more like religion. From what I can see there are more problems (toolchain, unstable, abi problem) so far than advantages (i.e. memory safety).

[1] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=281600

[no1wudi]

@cederom Thanks for your information !!

Currently, the support for Rust is mainly for application development rather than NuttX kernel development, so there might be fewer issues, mainly focusing on ABI compatibility.

Regarding ABI compatibility issues, they are not unique to Rust. Currently, we are troubled by this problem. In a large project, we have many teams with different permissions, so some code exists as static libraries. Therefore, any changes involving configurations or public data structures require synchronization between different teams and recompilation of these static libraries.

The structures required by the Rust standard library are relatively few and are all defined by POSIX (we should not make too many additional modifications to POSIX definitions), so there is a chance to achieve ABI compatibility under specific configurations. Overall, Rust has its issues, but it should be suitable as a starting point for exploration as a language for application development.

[yf13]

I am unsure if this is the same issue as the one I had with NuttX: when using NuttX in kernel mode, it is always a pain when app ELFs built with one kernel config doesn't work with another. The reason is that the .a files in nuttx-export-x.y.z.tgz package vary with the kernel config file.

Whether this is an issue depends on how we want NuttX to be:

• If we just want NuttX be an (invisbile) part of a single purpose embedded device, then likely it is not an issue.
• If we want NuttX to be a base of an eco system where different software can work together, then it is an issue.

The issue isn't specific to the app programming language as it happens to C as well.

[no1wudi]

I am unsure if this is the same issue as the one I had with NuttX: when using NuttX in kernel mode, it is always a pain when app ELFs built with one kernel config doesn't work with another. The reason is that the .a files in nuttx-export-x.y.z.tgz package vary with the kernel config file.

I guess for kernel build on a more powerful platform, maybe many apps runs at same time so it's a challenge to rebuild all of them once any config or change occurred in kernel side.

[cederom]

Yes I also believe in NuttX as alternative to bigger OS like Linux or BSD or even Android as we get more and more powerful MCU / SoC at single $ at some point we will need to have an OS where people can "just" build a package, install it, and it should "just" work :-)