Data corruption and loss at serial logging with OpenLog Artemis

[lksweber]

Hello,

I am struggling with serial logging the UART-output of my NEO-M8P GNSS module to micro SD. I want to log some ubx messages at 5Hz. I set the baudrate to 115200 to transmit some ubx packages at 5Hz (e.g. RAWX, SFRBX, PVT). After converting to RINEX in RTKLIB I noticed a lot of missing observations, sometimes more than half of the samples could not be converted. First, I thought it was an issue with the traffic on the UART. So I played around with the sample rate to 1Hz, kicked out some of the messages, incremented baudrate to 230400 and 460800, still the issue persisted. To fix it, I connected the Tx of my M8P to an oscilloscope and it looked good. In the next step I connected the UART to a terminal on my computer via an NS-RS232 to log it there and simultaneously used my OLA to log. Of the log on my computer, all samples could be converted to RINEX, of the log with the OLA only half of the samples. The file size for about 30 minutes of log was 19.807kB with the computer and 19.533kB with the OLA. Because the computer could log all the packets correctly at 115200baud, I assume this is an issue of the OLA. So for further troubleshooting I left the M8P aside and focused exclusively on the OLA:

Tools I used:

• NS-RS232 logic converter between UART (3.3V) and RS232, then to my computer. I checked it for different baudrates with terminal echo and transmitted a file between 2 of these, it worked fine. Also the logging of the M8P worked good. So I assume this is not the cause of the error
• Tera Term 5
• DiffMerge to compare files

I generated a binary hex file with numbered lines and random content to transfer via UART from my computer to the OLA to see what exactly is being logged there. On the left is the original file, on the right as logged by the OLA. This was at 115200 baud.

What really surprised me was after line 0007a130 where it logged the same sequence twice. This happens at multiple instances in the file.

At a baudrate of 9600 it was logged correctly but the OLA is supposed to work fine until much higher.

Your workbench

• Are you using a microSD card? If so, what size? How is it formatted (FAT, FAT32, or exFat)?
  Micro Sd-card of different sizes (16GB, 32GB, 64GB, formatted in FAT32 and exFAT) all showed that behaviour.

• At what frequency are you logging? For example, 10Hz, 1Hz, once every 5 seconds, etc.
  Serial logging at baudrates of 115200, 230400 all have data loss. GNSS rate at 5Hz

• What version of firmware are you using? This is shown at boot, for example: "Artemis OpenLog v1.10"
  Artemis OpenLog V10-v28

• How is OpenLog Artemis wired to your sensor(s)?
  From Tx of the computer to Rx on the OpenLog Artemis

• How is everything being powered?
  The whole setup is powered via the VIN port of the OLA with 3x AA battery cell. Same behaviour when powered via USB-C of the OLA

Edit:

Today I tried it again with different baudrates, the problem persisted. Also, I disabled everything else apart from serial logging and it didn't solve the issue.

What can be the cause for this issue?

Thanks a lot!

[PaulZC]

Hi Lucas (@lksweber ),

Thank you for reporting this. And thank you for all the detective work!

As I mentioned on the SparkFun Forum, I worry that you are asking too much of the OpenLog Artemis and that it simply can not "keep up" with the amount of data you are trying to log.

The serial data duplication is worrying. That suggests there is a flaw in the way serial data is added to the internal buffer, and written from the buffer to the microSD card.

Some questions and suggestions:

Are you only logging the serial data from the GNSS? Or are you also logging the PVT data via Qwiic - also at 5Hz?

Do you have any other sensors connected? Are you logging data from those also?

Are you logging the IMU data too?

Please try the OpenLog Artemis Serial Logger firmware. It will only log the serial data - it will not log IMU data or data via Qwiic. It is structured differently and uses separate threads to read the serial data and write it to microSD card. When I tested it (~3 years ago!) it worked well at 230400 baud. Please try 115200 baud first, then 230400. Do you still see the same data loss and duplication issues?

https://github.com/sparkfun/OpenLog_Artemis_Serial_Logger

The OpenLog Artemis has limitations. For your application, you may have to use different hardware. For the fastest logging rates, you may have to change to (e.g.) ESP32 with: the microSD card connected via SDIO (not SPI); and the GNSS connected via SPI (not I2C). I wrote an example for the SparkFun DataLogger IoT 9-DoF, which was able to log SFRBX, RAWX and NMEA data from the ZED-F9P at 20Hz:

https://github.com/sparkfun/SparkFun_DataLogger_IoT_9DoF/blob/main/Examples/Hardware_Tests/14_Log_GNSS_via_SPI_to_SDIO/14_Log_GNSS_via_SPI_to_SDIO.ino

There are other DataLogger IoT examples which you may find useful. They show how to log data from the on-board IMU and Magnetometer at high rates:

https://github.com/sparkfun/SparkFun_DataLogger_IoT_9DoF/tree/main/Examples/Hardware_Tests

I hope this helps.

Best wishes,
Paul

[lksweber]

Hi Paul!

From the GNSS I am only logging serial via UART, the Qwiic of the M8P is not connected.
I am using a BME 280 atmospherical sensor on the Qwiic of the OLA, and yes I log the IMU at 2Hz.

The reported issue occured also when only logging serial with the standard firmware (at baudrate of 19.200 already), without any additional sensors and IMU disabled. It seems it is a firmware issue that the serial logging has thes problems.

The serial logger firmware with the M8P worked good at 115.200 bauds, but this kinda takes out the fun when the OLA degrades to a simple serial logger instead of this nice thing with IMU and Qwiic sensor support.

When using the standard firmware as described before, the measurements get corrupted while logging and RTKLIB can not convert them to Rinex due to mistakes. Sometimes sequences in the question of minutes are corrupted until it works again.

What baudrates can you recommend with the OLA? I bought it because of the description where it said it can log up to 500.000 bauds, I am still way below that. What configurations do you recommend for logging serial and IMU simultaneously? My project requires this setup and for that the OLA seemed the perfect tool.

Best wishes!

[PaulZC]

Hi Lukas,

The honest answer is "it depends"...

The standard OLA firmware is written as a single thread. For serial logging, it relies on the Apollo3 hardware serial receive buffer not overflowing while the firmware is reading sensors via I2C or SPI and writing the data to microSD. If the hardware buffer overflows, data is lost. The data duplication is a separate issue, and I have never seen that before. There must be a flaw in the way the data is buffered while it is written to SD, but I can not think what that could be.

The hardware supports serial data rates up to 500000 baud. But being able to log continuous data at that speed is a very different challenge. Logging while also reading data from I2C and/or SPI is a different challenge too.

I am sorry, but the OLA may not be a good solution for your project. For fast data logging, there are much better solutions - ESP32 with SDIO is one. Or, if you want to keep using the OLA, you may need to write your own dedicated firmware. Perhaps you could start with the Serial Logging firmware and add the extra code to read and log the IMU and barometric data?

Things that may help:

• Increase the I2C bus to 400kHz. This will decrease the time taken to read the barometric sensor
• Disable the "Log to Terminal" option. Serial prints slow down the code too

Apologies,
Paul

[lksweber]

Hi Paul!
I reduced the update rate to 2Hz and checked the output of the M8P with an oscilloscope. The Tx is occupied by about a third of the time only when at 115.200 bauds.

I could not get the OLA to log UART correctly with the standard firmware at any baudrate above 9600, even when disabling every other function and increasing I2C to 400kHz. I cannot imagine it to be asked too much at 19.200 already, so the problem must be somewhere with my settings.

Is there a chance you can try to reproduce the problem by sending a test-file like shown in the initial post via UART from your computer to the OLA and log to micro SD? Until what baudrates did it work without loss or shuffles?

Thanks a lot for the help so far!