Thermal Calibration corrupts accelometers data to show wrong horizon roll

[MartinStokelj]

Describe the bug
After enabling thermal calibration and calibrating the accelometers we can see horizon off for about 15 degrees roll angle.
Autopilot has 2 accelometers, LSM303D,MPU6000 and it looks like thermal calibration for both of them is somehow calculated wrong (horizon showing roll for about 15 degrees and after level horizon calibration we cant calibrate anything more, always getting error wrong direction detected).
After removing LSM303D from the board the thermal calibration gave good result since horizon was almost in level after calibrating accelometers.
Tryed on FW version 1.10.0 and 1.10.1

To Reproduce
Steps to reproduce the behavior:

1. Set VTOL DeltaQuad airframe and enable thermal calibration
2. Do the thermal calibration
3. Calibrate accelometers and you will see wrong roll on horizon
4. Do the level horitzon calibration, the horizon is now ok
5. Do the accelometers calibration again, here we get error wrong direction detected

Expected behavior
Thermal calibration should not corrupt the accelometer data.

Log Files and Screenshots
[Params with thermal calibration done] https://we.tl/t-5RIMulHMc5

Drone (please complete the following information):
https://www.unmannedtechshop.co.uk/product/unmanned-pixhawk-autopilot-kit/
pixhawk 2 fmuv2
Stack: PX4 Pro
Vehicle: VTOL
Version: 1.10.0
Git Revision: 4f6faac2c8000000

Additional context
pixhawk mini and fmuv5 works ok

[dagar]

Do you have the thermal calibration log and the generated pdf?

[MartinStokelj]

Do you have the thermal calibration log and the generated pdf?

If you give me info how to create them I will.
We are doing calibration onboard with params.

[dagar]

I'm referring to the offboard calibration method described here. https://docs.px4.io/master/en/advanced_config/sensor_thermal_calibration.html#offboard_calibration

[MartinStokelj]

I'm referring to the offboard calibration method described here. https://docs.px4.io/master/en/advanced_config/sensor_thermal_calibration.html#offboard_calibration

I will make offboard calibration tomorow and give the new feedback.

[MartinStokelj]

I'm referring to the offboard calibration method described here. https://docs.px4.io/master/en/advanced_config/sensor_thermal_calibration.html#offboard_calibration

I made offboard calibration, the result is the same, all logs and params are here
https://we.tl/t-3FQiUReeao

Is it normal that gyro and acc ids are not in the same order/numbered
Line 26: 1 1 CAL_ACC0ID 1246218 6
Line 41: 1 1 CAL_ACC_PRIME 1246218 6
Line 855: 1 1 TCA1_ID 1246218 6

[dagar]

I made offboard calibration, the result is the same, all logs and params are here
https://we.tl/t-3FQiUReeao

For convenience: wetransfer-c9996d.zip

[dagar]

You have relatively huge offsets for the board rotation. Can you zero them and normal calibration (including board level horizon) again?

SENS_BOARD_X_OFF	12.146007537841796875
SENS_BOARD_Y_OFF	-13.161460876464843750

[MartinStokelj]

Yes huge board offset is set after level horizon, level horizon is bad after calibrating accelometers.
If TC is enabled, after calibrating acc I see horizon with 15 degrees roll angle, after level horizon calibration the horizon is ok, but I cant make compas or acc calibration, getting ERROR wrong direction detected.
I think something must be wrong when merging data of acc0 with acc1 since it helped when we removed acc1 chip on one of the boards that we have.

[MartinStokelj]

According to logs, the TC is probably added to wrong accelometer.
LSM303D showing wrong offsets when board is on the table.

[MartinStokelj]

After compiling px4_fmu-v2_default.px4 I saw that there is no TC_A_ENABLE param...
Reading the web I saw we need to build as px4_fmu-v3_default.px4 to get the TC params.
With new FW the calibration off acc,gyro was success, and after level horizon all was ok, so I think the problem is solved.

[dagar]

Thanks for testing @MartinStokelj, but it's not technically resolved until #14776 is fully reviewed and merged.

[MartinStokelj]

Hi,
today I was able to create new thermal calibration files and put it on the drone. After calibration of sensors I was not able to ARM the drone. Error "High Accelometer Bias".
If I disabled the acc1 and set acc0 as prime, I was able to ARM the drone.
CAL_ACC1_ID 2162698
CAL_ACC_PRIME 1114122

I have upload the files (Vtol_ser18AP_thermal_cal.params is done after ARM error)
TC_error.zip

[dagar]

Hi,
today I was able to create new thermal calibration files and put it on the drone. After calibration of sensors I was not able to ARM the drone. Error "High Accelometer Bias".
If I disabled the acc1 and set acc0 as prime, I was able to ARM the drone.
CAL_ACC1_ID 2162698
CAL_ACC_PRIME 1114122

I have upload the files (Vtol_ser18AP_thermal_cal.params is done after ARM error)
TC_error.zip

Can you attach a log from boot showing the attempted arming?

[MartinStokelj]

Here it is
debugConsole.txt

[MartinStokelj]

Here is ulg file, I thought you wanted console log...
ulg.zip

[dagar]

Here is ulg file, I thought you wanted console log...
ulg.zip

That only has the raw sensor data (SDLOG_PROFILE). https://logs.px4.io/plot_app?log=f573f21d-edc0-4b32-8707-d12fb88da724

We need to see the regular logging to verify all the corrected accel data going into the estimator looks good. If you're able to re-test with a full log please try current PX4 master (after #14784 merged as a fix for this).

[MartinStokelj]

Here is ulg file, I thought you wanted console log...
ulg.zip

That only has the raw sensor data (SDLOG_PROFILE). https://logs.px4.io/plot_app?log=f573f21d-edc0-4b32-8707-d12fb88da724

We need to see the regular logging to verify all the corrected accel data going into the estimator looks good. If you're able to re-test with a full log please try current PX4 master (after #14784 merged as a fix for this).

I have used PX4 master, did the sensors calibration again and after that arming went ok.
https://review.px4.io/plot_app?log=7d5b9a30-1618-4822-acee-e0dd52ea0dd3

[MartinStokelj]

Today we did 6 flights with VTOL and all went ok. Thermal calibration works, since we didn't had to calibrate sensors after each flight.

[dagar]

Fixed.