EKF baro hgt timeout - reset to baro

[Ahrovan]

Bug Report

• flight log : log
• Crash after switch to Mission mode in Waypoint 4.
  Altitude setpoint = 20
  after waypoint-3, Quad copter descent to approximately zero altitude in waypoint-4
  
• F450 quad copter frame + Ublox M8N + PX4 v1.7.3 Stable Release
• Tested with SITL+Gazebo and without any problem.

Logged Messages
WARNING[navigator] offboard mission updated: dataman_id=0, count=19, current_seq=0
230:07:53 WARNING[lib__ecl] EKF baro hgt timeout - reset to baro
240:07:56 WARNING[lib__ecl] EKF GPS data stopped
250:07:56 WARNING[lib__ecl] EKF stopping navigation
260:08:00 WARNING[lib__ecl] EKF baro hgt timeout - reset to baro
270:08:07 WARNING[lib__ecl] EKF baro hgt timeout - reset to baro
280:08:15 WARNING[lib__ecl] EKF baro hgt timeout - reset to baro
290:08:22 WARNING[lib__ecl] EKF baro hgt timeout - reset to baro
300:08:29 WARNING[lib__ecl] EKF baro hgt timeout - reset to baro
310:08:36 WARNING[lib__ecl] EKF baro hgt timeout - reset to baro
320:08:44 WARNING[lib__ecl] EKF baro hgt timeout - reset to baro
330:08:51 WARNING[lib__ecl] EKF baro hgt timeout - reset to baro
340:08:58 WARNING[lib__ecl] EKF baro hgt timeout - reset to baro
350:09:05 WARNING[lib__ecl] EKF baro hgt timeout - reset to baro
360:09:13 WARNING[lib__ecl] EKF baro hgt timeout - reset to baro

[xdwgood]

@Ahrovan I just quickly analyzed the log.
Look this :

Around 07:46, GPS data stopped updating.Before 07:46, the uav's Angle/position was tracked normally.
So I think the GPS module is falling off in the air. Do you check the GPS module after the crash?　:smiley:

[Ahrovan]

@xdwgood This problem occurred today with link log file...

[Ahrovan]

@xdwgood The QUAD COPTER suddenly loses its height (in this time I used Lidar lite v3 too)

[xdwgood]

@Ahrovan
yes,I found a problem with altitude control before landing(time:04:21).　see picture:

It may be due to the pressure effect near the ground.　So it didn't crash, right?　@MaEtUgR Is the speed and height error of the z direction before landing(time:04:21) acceptable?

@LorenzMeier The customer used the most recent PX4 v1.7.3 Stable Release.　So I don't know if commander's low stack problem needs your attention.　I also have a suggestion: the ｄｒｏｎｅ is not allowed to enter NAVIGATION_STATE_AUTO_MISSION before take-off.　

[Ahrovan]

@xdwgood Thanks . I tried to prevent crash with changing the flight mode to "Altitude Mode": for now, there is no solution to the problem, I agree with you for altitude problem. for today I tested again with this result:
review log

• checked possible battery problems (ok, no problem)
• checked possible range finder problems (ok, no problem)
• checked possible waypoint's problems (ok, no problem)
• calibrated all sensor again.

Altitude estimator show "GPS Altitude" and "distance sensor" descend in 2:00 - 2:20:


but "Fused Altitude Estimator" and "barometer Altitude" show ascend Altitude?!

The QuadCopter actually descent the Altitude, but the QGC shows a higher Altitude of the setpoint.

[MaEtUgR]

@Ahrovan To my knowledge by default first priority for relative altitude estimation are accelerometer measurements. And your accelerometer z-axis while not being unusable or cliping all the time has a pretty high noise level which is probably due to vibrations and how the autopilot is mounted on the vehicle.

The second default sensor to use is the barometer because 3D GPS altitude estimates are in absolute meters above sea level closer to the real value but the relative tracking of the drone movement is a lot worse. And in my eyes your baro is likely the thing that causes most of your problems. It has a crazy offset to the real altitude. I've seen logs with 100-1000 meters offset but yours shows >6000m while you're at ~1300m. Also the noise of your reported barometer measurements is higher than I would expect. I'm used to a noise level of maybe ~1m and in your logs it sometimes hits >15m...

• How is your autopilot mounted to the vehicle? Did you take any special care how it's isolated for vibrations? Is it loosely mounted?
  I would suggest to mount it not loose at all but sticked/clamped using high frequency isolating material like e.g. a thin layer of foam.
• Where is your barometer? Is it in a case? Is there any foam around the pressure measurement hole?
  That would help with fluctuating pressure maybe generated by the propeller airflow.
• What baormeter do you have? Any uncommon device?

Let's also shout out to @priseborough because he might have more ideas and spot where the problem lies.

[Ahrovan]

@MaEtUgR Thank you. I will review and report on the cases today.

• I used "Mount Anti-vibration Plate Shock"
• I used generic "Pixhawk PX4 Autopilot PIX 2.4.6 32Bits" board. inside it, is small foam
• My barometer is MEAS MS5611

[Ahrovan]

New flight with new Hex Frame (All Hardware changed, Just Pixhawk stay):
https://review.px4.io/plot_app?log=33bb60a6-ce79-47ed-8573-a101b60fae44

crash due:
EKF baro hgt timeout - reset to baro

[Ahrovan]

see ecl

	// check if height is continuously failing becasue of accel errors
	bool continuous_bad_accel_hgt = ((_time_last_imu - _time_good_vert_accel) > (unsigned)_params.bad_acc_reset_delay_us);

	// check if height has been inertial deadreckoning for too long
	bool hgt_fusion_timeout = ((_time_last_imu - _time_last_hgt_fuse) > (uint64_t)5e6);

	// reset the vertical position and velocity states
	if (hgt_fusion_timeout || continuous_bad_accel_hgt) {
...
		// handle the case where we are using baro for height
		if (_control_status.flags.baro_hgt) {
...
			if (reset_to_gps) {
...
				ECL_WARN("EKF baro hgt timeout - reset to GPS");

			} else if (reset_to_baro) {
...
				ECL_WARN("EKF baro hgt timeout - reset to baro");

			} else {
...
			}
		}

[priseborough]

@Ahrovan The is no EKF issue until after the vibration levels suddenly become extreme. Following plot shows IMU accel scaled to units of g and the vertical velocity and position innovations:

You need to fix your hardware.

[Ahrovan]

@priseborough It looks like you checked the acceleration after the crash?

[priseborough]

Then that means there was no EKF issue during flight.

[Ahrovan]

A sudden change occurs between the height source (between the baro and the gps), causing a sudden loss of altitude in the situation that has been raised.

[priseborough]

The baro output is inconsistent with other sensors. It looks like a faulty baro sensor was responsible.

[Ahrovan]

This problem occurred on several hardware, it seems to me incorrect. Because there is no problem in altitude flight mode. Except in navigation mode with gps.

[priseborough]

What autopilot hardware are you using?

[Ahrovan]

@priseborough Pixhawk 1 Flight Controller

I also tested with pixhawk lite.

[tkris]

@Ahrovan is your Problem solved ?

I had yesterday the same issue 2 times with my S500 Quadcopter (Pixracer Controller) - EKF baro hgt timeout - reset to baro. My Quad suddenly gained altitude in altitude or position hold mode... very shocking experience. I am running PX4 1.8.0 but i had the same Problem with Arducopter.

Everytime the S500 Quad would arm and fly nice in stabilized mode but when i switch to altitude or position hold the problem occurs.

Flight logs:
https://logs.px4.io/plot_app?log=7e17a3e1-b15f-47fc-93fe-29bde57b09db
https://logs.px4.io/plot_app?log=d98bfd03-f428-416b-9130-15a1bcc3a01b

Maybe someone can help me out. I don´t know if my pixracer is bad or my GPS Module. Today i want to try a soft mount method of the FC.

[priseborough]

@tkris This is intended for the reporting of SW defects, not resolving problems with hardware, installation and setup.

[Ahrovan]

@tkris @priseborough The change in the hardware of the FC does not have an effect on the problem and this problem occurs again(So that we tested with versions Pixhawk 1 lite and Pixhawk 1).
The latest change we have made in the work plan to solve this problem is to change the gps module and to strengthen the power connections of motors. Changing the FC damper did not affect the problem. We look forward to working with you to solve this problem.