GPS not found after boot, need to Stop/Start #9461
Comments
|
This Problem arose between 1.7.3 and 1.7.4 as far as I can tell. 1.7.3 behaves ok, whereas 1.7.4 and newer exposes the above behavior. The only change during this time I found on GPS was 03b8cd7#diff-99edb50b3a54c5fd16d92e7fb53eecd5 but CTS/RTS is not used and UART4 (the one in use) does not have CTS/RTS anyway. Strange though, this may be a mRo X2.1 Issue as the Port in use is UART4/ttyS3 which is per default not mapped (as per init.d/rcS): |
|
My first guess would have been 03b8cd7#diff-99edb50b3a54c5fd16d92e7fb53eecd5 as well.
It's (incorrectly) just not mentioned there, but ttyS3 is the default GPS port. |
|
@bkueng I'll do that. |
|
Tried installing Master by building it local, and it worked. GPS is recognized immediately and after Reboots also. Here is the VER ALL Printout from local built Firmware: Here is the VER ALL Printout when installing MASTER via QGC (GPS not found after Powerup or Reboots): |
|
@bkueng as it worked already after Uploading a local built master, I skipped the bisect. Anything else I could check? |
|
hmm that is strange. The diff between 873e036 and 8a114c2 is small and it does not look as if any of the changes could be related. |
|
@bkueng Add on: Indeed I just looked through the Issues filed that included a VER ALL Printout, and it seems that up to 1.7.3 the Build Chain was using GCC 5.x, where newer Revisions use GCC 7.x |
I just checked: |
|
I did some testing on this and it left me with more confusion... Took another UBX (M8Q) with the mRo Unit and it behaves identically as with the M8N. Tried then on a Dropix where it worked in all conditions, but then this may have been because of the difference of the Build Target. I took my older AUAV-X2 Unit and tried the same (this one is using the Default FMUv2 Target), and this behaved the same as the mRo Units. So I tried to find differences on the HW Schematics. Between the original Pixhawk (FMUv2) and the mRo Unit there is a difference on the UART4: The Original is a 3V3 <-> 5V Levelshifter, that was left out on the mRo, AUAV but Dropix as well... As the Dropix seems similar from the HW Perspective as the mRo/AUAV I really don't know why they behave so different with identical Firmware. I could only imagine that has been changes with Powering 5V Externals, that behaves differently on the various HW Platforms. Dropix has no Power Controller for the 5V Supply but just connects the 5V from the PowerInput selector, while AUAV / mRo has a Circuit that is controlled by VDD_5V_PERIPH_EN on PA8 of the STM32, but so has the 2.4.6 rev of FMUv2 just using a different IC to do it. |
|
@tops4u thanks for providing me with the HW. I was able to reproduce it and this is what I found so far:
diff --git a/src/drivers/gps/gps.cpp b/src/drivers/gps/gps.cpp
index f8755d8bc7..78a9328f55 100644
--- a/src/drivers/gps/gps.cpp
+++ b/src/drivers/gps/gps.cpp
@@ -788,6 +795,11 @@ GPS::run()
case GPS_DRIVER_MODE_ASHTECH:
_mode = GPS_DRIVER_MODE_UBX;
usleep(500000); // tried all possible drivers. Wait a bit before next round
+ close(_serial_fd);
+ _serial_fd = ::open(_port, O_RDWR | O_NOCTTY);
+ if (_serial_fd < 0) {
+ //...
+ }
break;
default:
@davids5 any ideas? |
|
@bkueng - I am trying to relate the late start of the gps driver fixing the problem..... I would scope the interface and ensure the power and signals are clean. Ensure the power is not having a slow start or being toggled at boot.
what do you mean by affect but does not cause? There is a high possibility it is memory corruption startup order would affect the memory layout. Add a startup script to override the one in FLASH. In it just start the GPS. Does it work every reboot? |
I don't have a clear picture on that yet. What I noticed is that with different toolchain versions, it's different how often the error occurs.
Yes that works.
I checked this value: I scoped the signal, this is the startup going into bad state:
|
|
Thanks for the investigation @bkueng |
|
@bkueng - Please add all the FMU serial interface signals to the capture. Please also confirm that the power was shut off at +3 Seconds on the first picture. |
Do you mean TX?
There was no power off at +3s. This was during normal power up. There was no reboot between picture 1 and 2, just |
|
@philipoe which board are you using exactly? @LorenzMeier not yet. @davids5 wanted to reproduce it himself. |
|
Hi @bkueng thanks for the Progress in this case. I appreciate a Workaround but I think we should also figure out the root cause for this and why it is happening on some boards while it does not on others and why it has something to do with the Buildchain. It came to my mind, that the cause may be related to the Powersupply since AFAIK mRo and AUAV do have a switchable Supply (by Command from FMU) to the UART Port while Dropix for instance does not (always Powered with the FMU). |
|
@LorenzMeier - I have tested this on a https://store.mrobotics.io/Genuine-PixHawk-1-Barebones-p/mro-pixhawk1-bb-mr.htm with a 3DR GPS. I am powered from a bench supply via the powermodule. I am not able to reproduce. I will need to know the exact versions of hw that exhibits the failure. Links to the FMU and the GPS would be super helpful. |
|
You can try this one. It's master from 16.5. (1.8. BETA). Did not find the other one I had. This should fail as well... |
|
@tops4u The start uptime seems to vary but it always started. I am repeatedly doing |
|
Did you Update the Bootloader according to the docs? Otherwise it would identify as FMUv2 |
|
Nope! My Bad. I just used it out of the box. I see that section....I just skipped to the pretty picture to hook it up. |
|
Ok. Even with the correct bootloader QCG is having issues finding the board. regardless installing your bin GPS is always coming up. @tops4u It sounded to me as if you were getting a hard failure. I have not seen that ever. Can you reduce your setup to just the GPS a Power module and the console? On the console issue 'gps status' every .25-.5 Seconds (I use putty right click) and see it it does start eventually. Is it true that no matter how long you wait your GPS never starts? Do you have a link to the actual GPS you are using? |
|
I use a Mac. Never had issues finding the Board. Tried with Drotek uBlox M8N / M8Q. They work like a charm on Dropix regardless of Build/Version. |
|
@davids5 the following binary fails for sure, it is f87fa91, GCC 7.1.0, and I tested with USB power. |
|
@bkueng - thank you! I loaded the elf and tested with only a) USB b) a debug cable and c) the GPS connected results are the same as noted above: Are you connected to QGC? I am using a isolated debug console - it is an ftdi that will not back feed power to the FMU while connected and FMU is not powered. - I tested with the isolation switched off and it still will not fail. But the it worth a try on your side. Connect the FTDI after the unit is powered. What I have noted is that after the GPS has been on for a while then I disconnect the FMU's usb and reconnect it to power cycle the GPS is ready sooner. This may be some sort of a cold start vs warm start if dT of the last fix is small and boot is quicker. Could that have clouded the testing? |
|
Build Time 12.6.?? Current Master has a fix for this. Did you try the Files above? |
|
Check your Bootlog above
|
|
Just created a Log from the AUAV-X2 (FMUv2) which suffers the same Problem. Picture of the Setup (most simple, a uBlox M8Q and the Console just GND, TX, RX): Log after issuing REBOOT (Issued "GPS Status" by typing it on the Term, so I guess it took 1-2 secs for each GPS Status request): Flashed 1.7.0 which worked fine: There was no change in Setup... just Update (downgrade) of the FW from Master (dated back from 6.6.) to 1.7.0 regular Release from the List of 1.7.0 Binaries. |
The files @bkueng sent have that time/date. I am not seeing the problem you are pointing out. What am I missing? |
|
@davids5 : First Log ist with Master from last week -> GPS fails after reboot. Second Log is with 1.7.0 from December where GPS starts on the spot after booting. Same Setup, same HW, etc just flashed older FW. |
|
@tops4u - I understand the content of #9461 (comment). The June 6 vs June 12 part (6.12) was where you had me confused from #9461 (comment) The #9461 (comment) test results was with FW @bkueng sent of his build that he has failing on the X2.1 HW you sent him. Are you thinking he sent me the wrong files? I placed his FW on the HW I have a picture above of. That works. You are also seeing the issue on your X2 HW with a different GPS and FMU than I am testing with. Things that may be different:
If I can not replicated it with my HW, I will most likely need the HW @bkueng has to track this down. |
|
@davids5 - Regarding the Questions:
I have not yet tested with Beat's FW but I can do this today evening. I have no spare mRo Unit so I tested with the older AUAV X2, the only mRo I have left is in a Hexa. Let me test this later today. |
To resolve the confusion: I rebuilt the firmware on that date. What's relevant is the commit hash, which is one commit before my work-around on master. So everything is correct. @davids5 I see you executed |
|
@bkueng I am not sure, build infrastructure is relevant. When I build master on my Mac with GCC 5.x it does work as well, only the Server Buildchain fails for me. The Toolchain you used is not identical to the one on the Server (Toolchain: GNU GCC, 7.1.0 vs. Toolchain: GNU GCC, 7.2.1 20170904 (release) [ARM/embedded-7-branch revision 255204]) Did your Build fail on your FMU? |
Yes |
|
@bkueng Ok, I'll check with your build on my mRo Unit later today. |
This is the root cause of #9461 The _pins array was initialized to -1. It was used to index the _gpios array. The value at _gpios[-1] was a number that mapped to Analog mode on Port A pin 0. These is the UART4_TX pin and was being reconfigured by the fault in the camera_trigger to an alaog input.
|
@tops4u, @bkueng - Thank you for your help! All of this make sense. All the weirdness: What compiler, how it was built would have changed the data indexed by the -1 on the gpio array. Why the reopen worked. The data at gpio[-1] passed to the px4_arch_configgpio would have affected some IO on the CPU. Yikes!!!. In this specific case UART4 TX PA0 was overwritten to an Analog input by the camera trigger code bug. |
|
Cool! Nice work everyone! |
This is the root cause of #9461 The _pins array was initialized to -1. It was used to index the _gpios array. The value at _gpios[-1] was a number that mapped to Analog mode on Port A pin 0. These is the UART4_TX pin and was being reconfigured by the fault in the camera_trigger to an alaog input.
This is the root cause of #9461 The _pins array was initialized to -1. It was used to index the _gpios array. The value at _gpios[-1] was a number that mapped to Analog mode on Port A pin 0. These is the UART4_TX pin and was being reconfigured by the fault in the camera_trigger to an alaog input.
Bug Report
My ublox 8M GPS is most of the time not found after boot. Sometimes Powercycling helps or issuing a GPS Stop / GPS Start command. Before the GPS is cycling various GPS Products but is not able to find the ublox. Is this HW related - I use a mRo X2.1. Just rebooting does not fix the problem, after each reboot the GPS is again not found.
Bootlog below:
The text was updated successfully, but these errors were encountered: