Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix TECS init when switching into a altitude controlled mode only once in air #22499

Merged
merged 6 commits into from
Dec 12, 2023

Conversation

sfuhrer
Copy link
Contributor

@sfuhrer sfuhrer commented Dec 7, 2023

Solved Problem

Fixes #22487 resp. https://discuss.px4.io/t/cube-o-px4-v1-14-0-erroneous-pitch-roll-setpoint-in-autonomous-mode/35557/11

The issue is only noticeable when the vehicle is switched into an altitude controlled mode when already in air, and there is no airspeed sensor connected. Then the airspeed state is 0, which leads to a division by 0 while calculating the pitch setpoint in TECS. And TECS initializes wrongly and constrains the INF only to a large value of 5rad.
It is NOT observable if at any time before takeoff the flight mode was already in an altitude controlled mode for a bit, then TECS is correctly initialized by the time it is actually needed.

Solution

ac33e4e is the real fix. Prior to this commit, the _control_params, eg. _control_param.pitch_max , are already used in TECS::initialize() but not initialized with the actual settings on the vehicle. As the default for the pitch min/max is 5rad the effect is then clearly noticeable.
e177e9a is patching the airspeed-less handling, which is further robustified with 4a8f346 and 0964831. Without these changes, the .tas state is at 0 which then leads to a division by 0, and due to the wrong pitch limits it doesn't get constrained properly.

The other commits are not directly related to the crash, but proposals for mini clean ups and further guards.

Changelog Entry

For release notes:

Bugfix: Correct TECS initialization when switching into an altitude controlled mode only once in air

Alternatives

Test coverage

SITL testing. The issue is reproducible in SITL. As the SITL though inits in Position mode, there is a small hack needed to not have it run TECS::update() before switching into Stabilized mode: do not run it within 10s of boot --> if (_local_pos_sub.update(&_local_pos) && _local_pos.timestamp > 10_s) . Further one has to disable the airspeed sensor, I did that by commenting out the publishing of the simulated differential_pressure topic in SimulatorMavlink.

With current main:
image

With this PR:
image

@DronecodeBot
Copy link

This pull request has been mentioned on Discussion Forum for PX4, Pixhawk, QGroundControl, MAVSDK, MAVLink. There might be relevant details there:

https://discuss.px4.io/t/cube-o-px4-v1-14-0-erroneous-pitch-roll-setpoint-in-autonomous-mode/35557/13

@sfuhrer
Copy link
Contributor Author

sfuhrer commented Dec 8, 2023

@KonradRudin in f57ae5f and following I propose to completely remove the manual TECS resetting from the fw_position_controller, and instead do the initialization only at one place anymore: internally in the TECS::update() when the dt is too large. This removes a lot of (seemingly) unnecessary complexity without a downside that I found so far.

Copy link
Contributor

@KonradRudin KonradRudin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice to make things a bit more robust. Added some remarks, some are just for discussion.

src/lib/tecs/TECS.cpp Outdated Show resolved Hide resolved
src/lib/tecs/TECS.cpp Show resolved Hide resolved
src/lib/tecs/TECS.cpp Outdated Show resolved Hide resolved
src/modules/fw_pos_control/FixedwingPositionControl.cpp Outdated Show resolved Hide resolved
src/lib/tecs/TECS.cpp Show resolved Hide resolved
@KonradRudin
Copy link
Contributor

@KonradRudin in f57ae5f and following I propose to completely remove the manual TECS resetting from the fw_position_controller, and instead do the initialization only at one place anymore: internally in the TECS::update() when the dt is too large. This removes a lot of (seemingly) unnecessary complexity without a downside that I found so far.

In general i think it would be better if the initialization is done outside (and inside only if we know that the timestamp would be unsafe due to CPU spikes or whatever), because it is easier for the executor to know, if it switches to TECS control loop or not. Internally we can only check this based on the timestamp. However, currently the fixedwingpositioncontrol has wierd logic on when to initialize it and it is convoluted as well. So your fix makes it cleaner

Copy link
Contributor

@KonradRudin KonradRudin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Have you tested this again in SITL with your changes to reproduce it? Also, can we assume that altitude and altitude_rate are always sane? (i guess so). Then you could squash some commits.

@MaEtUgR
Copy link
Member

MaEtUgR commented Dec 11, 2023

I almost crashed because of the underlying issue yesterday when I switched to RTL for a test:
Video: https://youtu.be/Y-m5iHEY-g0?t=81
Log: https://review.px4.io/plot_app?log=d4cc6eb1-2eea-4db7-aa8f-6605da76bf11

My analysis pointed exactly toward the things fixed in this pull request. It's reproducible in SITL make px4_sitl gz_rc_cessna when not publishing airspeed and taking off in Stabilized. I added logging of the TECS status also when it's not running and it got continuously updated by the initialize structure. That update always happened with zero airspeed and hence the calculation of pitch_sp_rad depending on the rest of the corrections is intermittently either -Inf or Inf and the cets constrained by ±5 radians pitch angle which corresponds to ±286° which makes the plane dump into the ground (like in my case) or stall trying to make a steep looping.

I got lucky with having enough altitude to react, switching back to Manual mode quickly enough and the plane surviving the >9g pull up 😅

If desired I can test these changes in SITL later today.

Signed-off-by: Silvan Fuhrer <silvan@auterion.com>
Signed-off-by: Silvan Fuhrer <silvan@auterion.com>
The control params (eg min/max pitch) are used before they are
correctly set by TECS::update(). While this is an issue we should fix,
it also doesn't hurt to set them to more reasobale values (eg 30° limit).

Signed-off-by: Silvan Fuhrer <silvan@auterion.com>
-remove external init, and instead always (but only) init when dt is too large
-init the controller params correctly

Signed-off-by: Silvan Fuhrer <silvan@auterion.com>
Signed-off-by: Silvan Fuhrer <silvan@auterion.com>
Signed-off-by: Silvan Fuhrer <silvan@auterion.com>
@sfuhrer
Copy link
Contributor Author

sfuhrer commented Dec 11, 2023

Squashed commits and rebased on main.

Looks good to me. Have you tested this again in SITL with your changes to reproduce it?

Yes I've tested it again, and it initialized properly now in all my tests (with and without airspeed, on ground and in air).

Also, can we assume that altitude and altitude_rate are always sane? (i guess so).

For now I would assume them to be sane. Otherwise we'd need to sanitize them on many other places. EKF2 hopefully invalidates the position estimate before it outputs garbage data.
Feel free to to propose additional sanitizations though where ever you deem necessary!

@sfuhrer
Copy link
Contributor Author

sfuhrer commented Dec 11, 2023

I almost crashed because of the underlying issue yesterday when I switched to RTL for a test

Sorry to hear, but happy that you had some altitude margin!

Copy link
Member

@MaEtUgR MaEtUgR left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for fixing this one 🙏

src/lib/tecs/TECS.cpp Show resolved Hide resolved
@MaEtUgR MaEtUgR merged commit 91ab09e into main Dec 12, 2023
88 of 90 checks passed
@MaEtUgR MaEtUgR deleted the pr-fix-tecs-init-main branch December 12, 2023 14:16
@MaEtUgR
Copy link
Member

MaEtUgR commented Feb 22, 2024

@sfuhrer JFYI I did some testing yesterday and RTL worked nicely exactly like expected in the exact same use case.
Video: https://youtu.be/Ssb4f8lfED8
Log: https://review.px4.io/plot_app?log=d3bd219a-664e-4428-9d94-002824303284

@DronecodeBot
Copy link

This pull request has been mentioned on Discussion Forum for PX4, Pixhawk, QGroundControl, MAVSDK, MAVLink. There might be relevant details there:

https://discuss.px4.io/t/help-debugging-auto-launch-crash-due-to-possible-tecscontrol-pitch-control-issue/36824/3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

v1.14.0 erroneous pitch/roll setpoint in autonomous mode
5 participants