PX4 1.6.5 Stbl: Loiter switching BUG.

[tubeme]

Hi guys,

We've had a total crash with a Skywalker, Plain Airplane profile. PX4 ver 1.6.5 stable.

https://logs.px4.io/plot_app?log=6daee88d-90d7-4cbe-abb7-440591fef87d

I think I found the reason but it is very strange and I need conformation. This is due to a BUG if I'm correct so please can you take a look and confirm the reason.

The pilot is not novice, it is pretty experienced one in terms of piloting a plane, but somehow sometimes the PX4 is overwhelming for him especially in certain situation and he is doing irrational switching (as many PX4 users might do).

This is the case:

He flew in Position and went in to Loiter (Pause). Then at some point he wanted to gain altitude while in Pause mode. In the graph it is obvious he started pitching up. But nothing happened as it should be, the plain continued to control the speed and throttle in the Loiter, despite the throttle lever was at 60%. In the latest implementation the Pause button will get the plane in Loiter from any mode. So initially he was in Position mode with activated Position switch. Absolutely normal operation as it is meant to be. Then strangely enough he switched to Manual in the attempt to gain altitude, without turning off the Pause button. The plane continued to circle, but strangely enough transferred the full control of the Throttle to the lever of the RC instead of controlling the throttle in the loiter itself automatically. At the same time the pilot continued to pitch up and gained altitude and I see it clearly in the graphics.. Because he had the lever at 60% throttle it was not enough to sustain the loiter, so the plane started stalling, but still trying to continue to sustain the loiter radius and gain altitude in the condition of under throttle. Because of the stalls the plane was rather going in spirals toward the pilot, and he explained that he thought it is drifting still not realizing what is going on. He did not realize that the Throttle is in his control. Then at the last stall the plane just plunged in nose dive. Because we have a hard limit of 45deg for the pitch, the plane did not have any altitude and time to recover.

I did not see any mechanical failure, neither to the servos neither to the GPS nor the airspeed sensor nor anything else.

Please can you confirm this and if it is a BUG it should be corrected.

Pause should be totally Auto mode, probably allowed to have some small amount of pitch control in order to gain or loose altitude, but in no case we have to give the throttle a manual control at this moment.

[LorenzMeier]

On first pass it looks like a mechanical failure: Everything is going well up until the point when suddenly the system is not able to achieve the pitch demand any more:
https://logs.px4.io/plot_app?log=6daee88d-90d7-4cbe-abb7-440591fef87d#Nav-Pitch-Angular-Rate

It pitches up instead of the commanded down pitch. The pilot is not involved in this (he is not controlling the plane at this point). Could you please check the plane and actuators to see if you see any signs of pre-crash mechanical failure on the servo or flight control surfaces for the elevator?

[LorenzMeier]

@dagar Happy to have your review as well.

[tubeme]

@dagar @LorenzMeier

Thanks a lot for the quick response. But I could not agree with the mechanical failure so I changed the name again. Here is why I don't agree.

Usually the first thing I do is look for mechanical failure. And after I exhaust all the options I continue to look for some other reasons. I spend at least 4 hours with this log. I haven't been at the flight, but I got very detailed explanation of what happened combined with a video.

1. After the crash we tested the servo. It is not a cheap servo (digital+metal gear) and despite the ploane was smashed on the ground with 30+ m/s the servo performs PERFECT. We will use it in another plane without any concern.

2. We are very pedantic on the wiring, using glue and tape to secure the PWM wires, so there is no chance we have a wire problem. The plane flew a lot with older versions of the PX4 1.5 without any problems. Long missions, long range etc.

3. Because of many many crashes due to servo failure I know exactly what the pattern of broken servo looks like. and I don't see it here. In 99.9 cases if there is servo failure we observe a total clipping of the MAIN output either to 1000us or 2000us. (The PWM output is giving extreme commands to compensate for the inappropriate behavior). There is no extreme PWM output from the controller (the most it was 1290us which is half the way from zero 1500us). Yes I saw the graph of the Pitch, and it is correct that it goes in another direction than the SetPoint but it is not extreme. In reality only what goes from the MAIN output can be used for conformation of mechanical failure. At first BTW I thought it is the mechanical failure of the pitch servo as well!!

4. The problem happens exactly at the moment when the pilot switched the main mode switch to Manual while leaving the Pause switch ON. May be it is another combination of events but again it is not mechanical.

5. At the same moment in time it is 100% obvious from the graphs that the Throttle is a straight line and not enough, compared to the throttle that was used in the Loiter. (there was 5 m/s wind which deteriorated the situation as well.) Also it is obvious that the throttle commanded to the MAIN output is exactly the same as the one coming from the RC!

6. It is obvious that the plane experienced 2 stalls which the PX4 recovered in the circle before the third one just plunged the plane in to the ground.

7. I saw the Innovation Check Bits for Yaw,Airspeed spike right before the plane plunged.

8. Lastly from the pilot I understand that there was no control of the plane at all! nither to the ailerons neither to the pitch neither to the yaw etc. If it is mechanical failure at least he could be able to control the Roll and Yaw which did not happen to the ground.

Please can you confirm it is a mechanical failure while digging deeper in the LOG.

Sorry I oppose you but I think this time it is not mechanical.

[tubeme]

@LorenzMeier The pitch demand could not be reached in the under speed condition as well...

[bresch]

@tubeme Are you sure the drone was in stabilized/manual during the spin? From the log it seems that it started in stabilized (not in position as you described) and then switched to Automode/Mission
edit: Ok, the pilot switched mode but was still in loiter. Does this works usually? (being in loiter and in manual control at the same time)

edit2: @tubeme It looks like it's in fully manual: the controllers are saturating but the actuators outputs are following the RC controls directly

[tubeme]

@bresch Exactly my point. It was funny Manual... It does not have to be like this!!! It is a corner case, when he made a switch in the modes while Pause was still ON.

Pause (Loiter) should be implemented like RTL.

Before that it was not possible because you had to switch to Mission in order to have the Pause activated. But after a long discussions I think @AndreasAntener made the Pause to work like the RTL and could be activated from all modes. I insisted on this implementation and it is just very good one. But probably there is something left in the implementation, and this is the first time anybody has this case.

This is due to inconsistent pilot. But that is what pilots do, sometimes they are inconsistent. In our other projects the pilots are pretty well trained on PX4 so they are very consistent. So we did not experience it yet.

[tubeme]

@bresch It happened after this issue: #6733

[tubeme]

It is very good idea to give a pitch control in the loiter in order to climb or descend but other than that there should be no manual control. Otherwise it happens this :) Total crash

[bresch]

@tubeme I didn't dig into the implementation of the modes switch, but obviously this is a bug and has to be fixed. There is no reason that it goes into fully manual if you switch to stabilized.
However, with the current architecture, I don't think we could nicely implement a hybrid version with stabilized pitch control while having the rest controlled by the automode in loiter. Altitude manual control might be feasible but would still require some further effort. What is you opinion @dagar ?

(Even if it's unlikely that someone has the same problem soon (I don't think many people use loiter on a 2nd switch), but I set it to Priority critical since it leads to an unexpected mode that can easily lead to a crash.)

[tubeme]

@bresch Fair enough let it be Full Auto mode then, as it is meant to be for now.

There is another approach to the problem. If it is in Pause, then the RC pitch up command could trigger MAV command for altitude gain or descend and not manual controlled one.

[dagar]

Coming in late, but I see 2 separate things here.

1. Something went wrong with the vehicle while it was loitering. First it started gaining altitude ~10:20, then it stalled and started diving. This seems to start approximately when the pilot flips the mode switch (pink line), but appears to be a coincidence. Look at the throttle, commanded altitude, airspeed, pitch, roll, etc when the switch happens.
   

Look at the airspeed, altitude, and pitch actuation. The vehicle started climbing despite significant negative pitch actuation (light gray) and cutting throttle (yellow). Something physically happened to the vehicle. The vehicle stalled and the pilot briefly switched into STABILIZED mode (unintentionally?).

2. The main mode switch (first plot pink line) did not change the mode. The vehicle stayed in loiter as intended. Later the pilot changed the loiter switch (first plot dark red line) and briefly placed the vehicle into STABILIZED mode (3-4 s) before going into RTL during the dive.

If there's going to be any work on mode switching we first need to unify the old "advanced" configuration with the new "simple" configuration. We simply don't have the resources to maintain multiple versions of everything.

EDIT: Controllers were still running, but the px4io was ignoring actuator controls

[dagar]

Standby, I may have found something.

[dagar]

Sigh, this is the IO taking over in MANUAL (main mode switch), and not aware of the FMU side logic change respecting LOITER switch precedence. The controllers are still running, but don't know their output is ignored. Hence it looks convincingly like a mechanical failure.

This is why we need everyone focused on using/developing/testing the same core feature set. I'll look into a short term fix for v1.7.0.
@acfloria FYI

[tubeme]

@dagar I know he commanded RTL because of his explanation, but i see it was too late in the dive, so no time for recovery...

Probably we could omit the old style in favor of the new style. But the current new style with only 6 positions (that I can split in to multiple switches in the RC mixer) is not enough for more complex setups regarding the PRO use, being pro rider use (multiple of acro manual ratitude modes) or pro commercial use like geodesy or observation and inspections use.

There was discussion on this matters I remember in the issue i mentioned above : #6733

We could rethink the whole idea with a new perspective, make it simple and tight but at the same time flexible and powerful.

[dagar]

Yes you can see the brief RTL in the my first plot (light red near the end).

[LorenzMeier]

Fixed in the referenced PR.