Updated Sensor API

[microbuilder]

API meeting (@carlescufi):

• Currently (for the next month, until 8th March) under discussion in this Google doc: https://docs.google.com/document/d/1UsElJJHRudeWXb-tYwXA6qt9gF6PH8wwS7PjhqO1MLI/edit#bookmark=id.jkiwnksvdslt

Updated Sensor API

This issue is an attempt to start identifying improvements that can be made to the current sensor API in Zephyr, with the goal of defining a new API that will eventually replace the existing one.

A more formal proposal will be made in due time, but any weaknesses in the current API can be listed here, or requirements for a new API.

The current proposals are based on existing issues and discussion during the weekly API call, and are subject to revision. They do no reflect definitive positions, rather a starting point to define a more flexible sensor architecture.

Project Management

The overall sensor API project is tracked here.

Components (Issues, PRs)

The following issues or PRs are part of the overall sensor API project, and are used for tracking purposes:

• TODO

Two-tier sensor data API (raw + SI)

• Raw data at the lowest level (mandatory: channel 0, attribute 0):
  • Memory efficient (Two mandatory 32-bit values today)
  • Low latency
  • Higher processing burden in user land
• Standardised SI-based data at the higher-level (mandatory: channel 0: attribute 1):
  • Higher per-sample memory use
  • Higher latency
  • Low user processing burden (standard SI units and scale, filtering, etc.)
  • Smarter event callbacks based on richer data filtering, etc.

Both tiers must be implemented in a valid sensor driver.

General Requirements

Timestamp API

• Timestamping API for individual samples, required for DSP filters, logs, etc.
• An option should be provided to enable or disable this feature (memory constraint).

Flexible Execution Context

• Allow execution context to be specified by user, with a goal of enabling
  a single thread or limited thread count for all sensor activity.

Internal FIFO Management

• Sensor data is cached via a FIFO buffer associated with channel attributes
• Allows more efficient resource usage, lower user-burden.
• FIFO should have watermark/threshold alert(s).
• An appropriate set of helper functions should be provided to filter and manipulate buffers.
• It should be possible to use one buffer across multiple attributes efficiently.

Sensor Data

• Enable raw data for smallest footprint with higher user burden.
• Driver MUST implement raw->stdtype conversion function(s) or provide necessary converstion coefficients (via attributes, etc.)
• Determine sensible SI unit/scale per channel or sensor type
  • Should scale be adjustable at the KConfig level? Auto-scaling?
• Where possible, flag missed sample(s) event between reads.
• Optionally associate variable meta-data (die temp, etc.) with sample.
  • Memory an issue here
  • Seems like this should possible via attributes

Current thinking is that all sensor data will be based on attribute records, with each channel in the sensor driver having appropriate attributes like CHAN_ATTR_RAW_DATA and CHAN_ATTR_SI_DATA. Each attribute is associated with an appropriate data type and payload.

Channels

• Channels indicate a specific type of sensor data or information (ex. CHAN_ACCEL_VECT, and may have generic catch-all versions like CHAN_LIGHT as well as more narrowly defined subtypes like CHAN_LIGHT_IR).
• Allow multiple instances of a channel.
• Meta-data can be assigned to channels via channel attributes, for example you can associate a wavelength value or range to an instance of the CHAN_LIGHT channel via the CHAN_ATTR_FREQUENCY attribute.

Attributes (Device + Channel)

A flexible, generic system is required here, though further discussion is
needed to define the initial attribute list.

Attributes will allow a high degree of flexibility, with only a minimal set of mandatory attribute required for basic drivers.

• Current thinking (subject to debate!) is that two types of attributes makes sense: device attributes, and channel attributes. Needs testing to validate of course.

Triggers

Triggers are associate with channel attributes, and provide a mechanism to indicate how or when a channel attribute should be updated.

• Review and revise current overly-narrow list of trigger events.
• Use a singly-linked list of callbacks that can be traversed asynchronously,
  starting in the IRQ handler but with the blocking SPI/I2C/ADC calls
  elsewhere. An slist allows this to be dynamically adjusted at runtime.

Events

• A callback based 'event' system might make debugging and characterisation easier by making it easy to defer error handling, and state tracking to the app level. For example, by having events fire and optionally subscribing to them via a generic callback, you can track conversion start and end events and see if there is a disparity (indicating conversion errors), etc.)

Additional APIs or Requirements

Calibration API?

• Common API to calculate, load/store and apply calibration coefficients?

Logging/Storage API?

• Centralised logging API for sensor data to make storing and off-loading
  sensor data at a later date easier? Persist data in a standard format to
  NVM, etc.?

Device Simulation?

While perhaps not restricted to the sensor API, a design goal should be
enabling simulation of specific sensors when HW isn't available, or specific
edge cases are difficult to reproduce on real HW (to properly test algorithms
or application code that relies on sensor data).

• Allow a sensor driver to 'register' itself at a specific I2C address, and
  in a simulated environment when a generic I2C read/write request is made, the
  I2C handler can redirect the request to the simulated function in the driver.
• The following functions would be required for I2C sensors, for example:
  • Read request handler
  • Write request handler
  • Init function to register the simulated device with the generic I2C layer.

For an example of this using I2C, see mcu_sim_i2c.h from Apache
Mynewt, and tsl2591_sim.c for a partial implementation of
this concept.

Permissions API?

Is a permissions API required/useful/relevant? Should some level of control be put in place over, for example, being able to access raw or sensitive data?

• Perhaps an 8-bit value can be associated with a device or channel attribute with flags like Read
  , Write, Public, Private, etc.?

Firmware Management?

Certain classes of sensors today have user updatable firmware. It's worth at least considering some of the implications around this, and how to update sensors on deployed devices, even if it's a very manufacturer-specific requirement.

Security Considerations

There are a number of critical security consideration to keep in mind when working with sensor data, and how it is exposed. For example, on devices with touch screens where a password may be entered, and where sensor data like accel/mag/gyro readings is publicly being streamed out, you can detect passwords based on the sensor data.

There should perhaps be some security mechanisms in situations like entering a password, where certain classes of sensors can be disabled to reduce the overall risk (disable the accel and gyro while the password screen is shown or while a keypad is being used for password or ID entry).

This fits into the permissions API idea to an extent, but is sufficiently different to merit individual consideration.

The W3 Generic Sensor API highlights a number of security risks associated with sensors, and some suggestions on mitigating these risks. It's worth reading through for the numerous practical implications the document details.

Discussion of accelerometer abuse in mobile devices: https://www.mysk.blog/2021/10/24/accelerometer-ios/

Sensor rate limiting on Android as a security mitigation: https://developer.android.com/guide/topics/sensors/sensors_overview#sensors-rate-limiting

Related Issues

#14008 : WIP: Sensor API Playground
#13455 : Improvements to drivers/sensors for light sensors
#1387 : Lack of a performant sensor API
#14245 : Sensor API channel definitions

Existing APIs to consider

• Linux IIO
  • Linux IIO standard attribute list
• W3 Generic Sensor API
• Android Sensor API and ContextHub
  • Related Android Sensor API components:
    • ISensors
    • ISensorManager
    • ContextHub
    • Channels, event, etc.: sensors-base.h
    • Core structs like sensors_event_t sensors.h

Context hub is intended to enable the use of additional co-processors for sensor data, and the interaction between context hub and the main kernel or sensor API might have some useful insights to consider.

[teburd]

I think looking at the linux IIO subsystem and how it works would be very useful in designing a performant and usable API for both drivers and applications.

Some nice things IIO does from my day exploration into it

• Static list of channels per device driver, allows multiple channels of the same kind!
• Channels may be read or write, providing not only input but outputs via DACs or other such devices.
• Attach buffers to channels with a pollable buffer, may be circular or not.
• Timestamps are optionally there per sample
• Direct DMA transfers to the buffer are possible to/from the devices, avoiding unnecessary copying. I think this is really a key point in zephyr land, as it would reduce memory usage and improve performance by offloading work to the hardware when possible. This in addition to perhaps a sparse timestamp of samples would allow hardware FIFOs to be used with I think very good results!
• Each channel and device has a set of attributes, accessible by char* names, which may be manipulated. Attributes themselves may be attached to individual or all channels, such as sampling rate and sample scaling.

Since each device describes its own channels with a static array of structs channel descriptors, problems relating to the current API limiting channels to 1 one of each kind is no longer a problem.

I would propose that in Zephyr to obtain the performance and flexibility desired several things would need to be adapted.

• A buffer like Linux kfifo.h which provides a fixed size ring, allows for direct dma transfers to or from it, as well as provides a method of polling for changes. Secondly provides a type agnostic API thats still efficient. The current ringbuffer in zephyr assumes a 32bit or 8bit type it seems, I don't think thats appropriate for reading from sensors where many channels may be interleaved. The less work the kernel has to do the better. An relatively easy way to do a DMA transfer from a SPI/I2C device into a buffer.
• A channel description struct that provides a type, units, range, byte order, byte offset when interleaved and the sample size may be anything from 1bit to 32bits per channel. Similiar to IIO.
• A device and channel api that allows manipulating attributes, including things like trigger sources, sample scaling, sample rate, and channel enable/disable.

So in zephyr I imagine something like the following be available in application space, I chose zio for a prefix, but it could be anything

ZIO_FIFO_STATIC_INITIALIZER(my_zio_fifo, 1024, false);
struct k_poll_event events[1] = { K_POLL_EVENT_STATIC_INITIALIZER(K_POLL_TYPE_ZIO_FIFO_DATA_AVAILABLE,
                                                                K_POLL_MODE_NOTIFY_ONLY,
                                                                &my_zio_fifo)
};

int main() {
struct device *dev = device_get_binding(MY_MAGN_SENSOR);
zio_set_attribute_int(dev,  ZIO_SAMPLE_RATE, 100);
zio_set_attribute_int(dev, ZIO_SAMPLE_SCALE, 4);
// these channels are statically defined and documents for whatever driver MY_MAGN_SENSOR is using, ex: lis3mdl
zio_channel_enable(dev, 0);
zio_channel_enable(dev, 1);
zio_channel_disable(dev, 2);
// at this point some default trigger source defined via DTS/K_CONFIG is enabled and handled
zio_attach_buffer(dev, &my_zio_buf);
// k_poll waits for new data to be available and our application code can deal with it as it comes in
// I imagine the fifo contains a compact representation of the sensor data, optionally with an associated
// timestamp.
// in the case of a device fifo being polled from, nothing really changes in the user interface here.
// in the case of a dma transfer happening, nothing in the user interface changes here.
int rc = k_poll(events, 1, 1000);
....
}

[microbuilder]

Thanks for the feedback. I agree with most of the comments above personally, and I'll dig into IIO a bit on my end having never worked with it. This doesn't address the Features API idea, which I think needs to be addressed, but it definitely seems like a good starting point architecturally.

[pabigot]

I'm generally in favor of re-using existing architectural solutions like IIO, but I'm more in favor of starting with a clear understanding of stakeholder needs. It may be that IIO is based on assumptions that don't apply to the application domains that Zephyr's intended to support.

My primary application is low-power sensors. I always know what sensor (HTU21D, BME280, SDP810, ...) is providing the readings, so I really don't care about the ability to ask for a temperature or a pressure independent of what provides the value.

I want to collect at a relatively low frequency (at most 1 Hz) or on detected change, so while some DMA features might be useful if they get in my way they're not wanted.

I want a single observation structure that provides a space-efficient representation of all the measurements produced by that specific device, so I know (for example) that the relative humidity I'm using was captured synchronously with the temperature I'm using, or that the three axes of acceleration were from the same observation. My application will know whether the pressure is Pa in a uint32_t or cPa in an int16_t.

I want notification of relevant events: new observation, threshold change, calibration completion. At this time I want that through a callback invoked from the GPIO or alarm interrupt. I'll decide if I want to do the handoff to the application through kpoll, kwork, or simply setting a flag to be processed by the main loop, and whether I need to synchronously do some other change (such as reconfiguring the interrupt management or triggering another observation). If I need to timestamp a captured observation I'll do that myself, since I know what clock will provide the precision and range I'll want to use for the specific application.

I want full control of the sensor's specific capabilities. Proper use of a CCS811 involves tracking and restoring baseline values at various points in the device and application life cycle, as well as updating stored environmental conditions. That's not going to be supported by a generic API in any usable manner. See #11993, as well as #12056 which it motivated.

In short, I think Zephyr's too hung up on providing cross-platform/cross-device API. All that abstraction costs code, RAM, and power, and I believe many embedded use cases won't want to pay for it. Perhaps I'm wrong.

My next/sensple branch has a primitive but functioning solution consistent with these requirements.

[teburd]

I actually disagree somewhat on the Feature API, Features per device could be done via named attributes. It would require some devices providing their named attributes in a public header. In some scenarios there might be additional channels of information that can be subscribed to as well, which themselves allow the device user to take advantage of those features.

Sensor Fusion example might be AHRS channels that are available but disabled by default. Enabling one, enables them all (they are tied together). The driver is responsible for reconfiguring the device with sensor fusion enabled and filling the buffer appropriately.

In a similar manner gestures might be a channel of enumerated data with a timestamp given. So again additional channels of information are available from the device, like I imagine numerous TDK 9 axis sensors, which are feature rich IMU chips, might have the following channels...

• die temp (1 channel)
• accel x,y,z (3 channels)
• gyro x,y,z (3 channels)
• magn x,y,z (3 channels)
• aux (5 channels)
• ahrs (3 channels)
• gestures (1 channel with int values describing gesture)
• pedometer (1 channel, with a timer trigger)

Now enabling/disabling these channels provides nearly all the functionality of the chip available with a pollable buffer and optionally timestamps, by enabling or disabling those channels.

Certainly there's some attributes for groups of these channels that should be mutable in some way.

• Gyro, Accel, Mag filtering settings
• Gyro, Accel, Mag range settings
• Gyro, Accel, Mag rate settings
• Aux (I2C master) register settings
• Gestures to notify of

Those could be attributes available on a channel or set of channels. Commonly named ones could be defined in a common header. Device specific ones would have to be exported by the driver in a different header, which is perfectly sensible.

[teburd]

@pabigot I think using name, typified attributes solves most of those problems. I agree, notification of events in a convienent and compact manner is important. Doing things in a callback is ok, but in Zephyr the context of the callback matters and that's partially why there's the trigger options now at compile time for k_work or k_thread callback contexts. Providing a callback in the IRQ context leads to problems in my experience as I experimented with that. I think channels that are pollable are quite workable, would allow the kernel to sleep saving power, and provide all the potential event notifications we could want. Better still the driver is the one dealing with reading spi/i2c registers on a trigger rather than application code. You get what you want at the end rather than needing to call back in to the driver to fetch data. How that data is fetched now is partially why handling triggers in the context of a IRQ handler really isn't feasible and why the current API is not performant or timely.

[microbuilder]

@pabigot I think this highlights why we need a two-tier approach to sensor data. There are certainly many occasions where you have full knowledge and control over the hardware you are using -- gain, integration time, etc., are known -- and memory efficiency is primary, in which case you should opt for a RAW channel and interface. I would propose that all sensor drivers, for example, should have a mandatory channel 0 that is always the raw data.

But there are situations where a high level of abstraction is definitely useful, especially in networks of long-lived devices. A key problem in the embedded space is obviously parts availability, and sensors that may be available today may be EOL, or with long lead times later, meaning I need to substitute in a replacement part to ship a new batch of devices. A high level of abstraction makes it relatively easy to replace accelerometer A with model B, C, or D with very little overall impact on the end node itself, or on the logging and management systems above it. I can continue to query 'temperature' from the higher level remote management system, with little concern for the source of that temperature info at the individual device level.

Both approaches are necessary and valid, but in my own case I generally am concerned with reasonably large sensor networks and collecting compatible data points that can easily be statistically analysed, etc., once a critical volumes is reached, and sensor availability is certain to change over time. The extra overhead of so much 'cross compatabiity' is very valuable to me, as is 'fast and efficient' access to raw data when I need to access something like accelerometer data at 1kHz, or something similar.

[teburd]

cross compatibility and high performance are also important to me, as is the ability to reconfigure the device at run time into various modes of operation for power savings and different triggers

[avisconti]

This thread is really interesting to me. I am currently exploring the possibility to add FIFO capability to one of ST sensor (LSM6DSO), but there is a significant lack in the sensor APIs. I agree with all the requests listed in the @microbuilder description.

I am particularly interested into following stuff:

• timestamp management.
  Currently there is no way to pass timestamp to application, but this should be a task
  performed by driver.

• FIFO management
  Absolutely mandatory.

• Virtual sensors handling
  This request seems to match with 'features API' in the description.
  We should be able to handle step_counter/step_detector/tilt/motion/orientation/pickup and so on virtual sensors

• Event management
  Sometimes it is necessary to report events to application (e.g. step detection).
  Propably triggers is the way and we just need to populate the list of possible 'trigger events'.

• sensor calibration
  I would need accel/gyro/mag calibration algo.

EDIT:
@mariotesi was suggesting also the following stuff:

• capability to upload a firmware to the sensor.
  This may seems a little bit akward, but we have some of the latest sensors that can be programmed with new functionalities.

• capability to dynamically probe features (virtual sensors) at runtime by app
  An application may get list of supported features from a sensor driver

• pass rotation matrix to driver?

EDIT2:

• set power modes (low power, high performance, ...)

• Self Test control (start, stop, check)

• Sensor Data Injection (for testing)

[avisconti]

Concerning already existing code I think that linux IIO framework is good, but I also suggest taking
a look to Google contexthub, which is an Apache 2.0 licensed work:

https://android.googlesource.com/device/google/contexthub/

[microbuilder]

@avisconti Thanks for the heads up on contexthub. It's also worth listing libhardware which contexthub relies on, and is a fairly mature API at this point considering the number of sensors it has had thrown at at:

• https://android.googlesource.com/platform/hardware/libhardware/+/master/modules/sensors/
• Channels, event, etc.: sensors-base.h
• Core structs like sensors_event_t sensors.h

This was the basis of the Adafruit unified sensor API years back, but it has definitely evolved since I first looked at it ages ago.

[avisconti]

@microbuilder
Yes, I think that libhardware is the contexthub counterpart in Android which run on the AP, while contexthub itself is a firmware that runs on a companion micro.

But, yes, all that work is pretty standard now and runs on many Android based mobile phones.

[avisconti]

Anyway, I personally think that this new sensor API related work should be scheduled with high priority for v1.15.

[microbuilder]

An initial proof of concept repo has been created here -- along with some useful labels and issues -- to enable collaborative development until this is ready to advance to a proper PR and RFC in the zephyr repo: https://github.com/microbuilder/zsensor

topic-sensors branch created, project management tracking pending.

[avisconti]

@microbuilder
I think that, as a first step, we would need to reconsider the data structures associated to sensor.
Currenty everything is just a senor_value with val1 as integer part and val2 as fractional part.

Looking at google sensorhub I see that there are more complex data structures.
Take a look to https://android.googlesource.com/device/google/contexthub/+/refs/heads/master/firmware/os/inc/sensors.h

Data can be of folloqing types:

• NUM_AXIS_EMBEDDED
• NUM_AXIS_ONE
• NUM_AXIS_THREE

Embedded is used for algo data, like step_counter data, tilt/sig_motion events, gesture events and so on. Axis_one for things like environmental (pressure, temperature, humidity ...). Axis_three for accel/gyro/mag/...

Each one of the 3 types has a different structure, with timestamp if it is the case.
Every sensor at a given time generates an array of this samples, which may be of size one or
more than that (in case for example of FIFO enabled).

[microbuilder]

@avisconti Sorry that this kind of spans a couple PRs and issues at this point, but you can see some initial thoughts in code here, along with some other ideas by @BFrog earlier: #14008 (comment)

I agree the current system is extremely limiting, though!

[jgl-meta]

@MaureenHelm @microbuilder Can you provide a status update for this task? Will this be ready for release 3.4?

[teburd]

I believe much of this will have been resolved with the inclusion of #60063 while more still will come from the sensing subsystem

[microbuilder]

@teburd Agreed. This is already ancient, and I think any future work probably deserves a new RFC. Thanks for the work put into this so far, along with @yperess.

[teburd]

I think this should remain open until #60063 is fully merged, as until then its still very much a valid thing.

[yperess]

#61022 resolves the portions of this issue about multiple channels of the same type.