Configuring receiver -> then UBXReader stops reading after several succesful readings (F9P)

[djdan7]

Firstly I want to thank you for the effort you put into creating this library - it's AWESOME and you are GREAT! :D

My config:

u-blox C099-F9P // FWVER = HPG 1.13 // PROTVER = 27.12
navigation frequency: 10Hz
connection via USB cable (UART1)
windows 10 x64
python 3.7.9
pyubx2 ver. 1.0.2

My problem:

I want to start GNSS (via CFG-RST message), then read UBX messages for some time (in my case these are MON-RF, NAV-DOP, NAV-EELL, NAV-HPPOSLLH, NAV-PVT, NAV-SIG, NAV-STATUS) and stop GNSS (again via CFG-RST). The reason why I want to stop and start GNSS is that I want to save battery while keeping GNSS parameters in memory (so then I may perform hotstart).

How my code below works:

#receiver is stopped
-Starting GNSS is succesfull
-Reading several first message is also succesful, but then it hangs up and wait for message from serial (but the receiver is still working, and the serial send messages)
-I terminate the script
#start again (now the receiver keeps working - we are not stopped it)
-"Starting" GNSS is succesful - it just sends the message, but of course receiver is already wroking
-Reading all messages is succesfull
-Stopping GNSS is succesful
#again receiver is stopped
-Starting GNSS is succesfull
-Reading several first message is also succesful, but then it hangs up...
...

I tried to set validate flag to True - it raise

UBXStreamError(f"Unknown data header {byte1}. {nmeawarn}")
pyubx2.exceptions.UBXStreamError: Unknown data header 

What I noticed - after several OK readings it starts to divide binary message to single bytes and this is why this error occurs.

To prevent this I need to close port after writing, sleep for 1 second (shorter values don't help), open serial port again and flush it. This way everything works OK. But the problem is that in the future I would like to configure receiver on the run and then 1 second break looks ugly :/

Code:

import time
import serial
from pyubx2 import UBXReader, UBXMessage, SET

print("hello")

ser = serial.Serial(port="COM9", baudrate=921600, timeout=1)
print("Port is open? -> ", ser.is_open)

if ser.is_open:
    # START GNSS
    msg = UBXMessage(6, 4, SET, resetMode=9)
    print(msg)
    ser.write(msg.serialize())
    print("STARTED!")

    """
    # CLOSE, SLEEP and OPEN AGAIN
    ser.close()
    time.sleep(1)
    ser = serial.Serial(port="COM9", baudrate=921600, timeout=1)

    # PORT FLUSH
    print("dummy read", ser.read())
    ser.flush()
    ser.flushInput()
    ser.flushOutput()
    """

    # READING UBX
    i = 0
    while i < 300:
        print("reading ", i, ":")
        ubr = UBXReader(ser)
        (raw_data, parsed_data) = ubr.read()
        print(raw_data)
        print(parsed_data)
        if parsed_data.identity == 'NAV-HPPOSLLH':
            print("LON = ", (parsed_data.lon + parsed_data.lonHp/100)*10**(-7))
        print("\n")
        i += 1

    # STOP GNSS
    msg1 = UBXMessage(6, 4, SET, resetMode=8)
    print(msg1)
    msg1 = msg1.serialize()
    ser.write(msg1)
    print("STOPPED!")

    ser.close()

else:
    print("Problem with serial port")

[semuadmin]

Hi djdan7. Glad you're finding the library useful, but sorry to hear you're having problems.

I notice in your code that you're instantiating the UBXReader object repeatedly within the read loop (and thereby creating 300 separate instances). This only needs to be done once at the head of the loop, i.e.

# READING UBX
    i = 0
    ubr = UBXReader(ser)
    while i < 300:
        print("reading ", i, ":")
        (raw_data, parsed_data) = ubr.read()
        print(raw_data)
        print(parsed_data)
        if parsed_data.identity == 'NAV-HPPOSLLH':
            print("LON = ", (parsed_data.lon + parsed_data.lonHp/100)*10**(-7))
        print("\n")
        i += 1

Suggest you try this change first and see if it resolves your problem.

You could if you prefer also use the UBXReader iterator with a millisecond timer, rather than a fixed number of iterations. e.g. something like this (pseudocode):

duration = 1 minute
start = now()
stream = Serial(port, baud, timeout=5)
ubr = UBXReader(stream)
for (raw, parsed) in ubr:
   if rawformat:
      print(raw)
   else:
      print(parsed)
   if now() - start > duration
      break

Hope this helps.

[djdan7]

Thank you for pointing out that I've wrong instantiated the UBXReader! It didn't solve the problem, but the code is now better :D

I will also answer to your previous suggestions:

1. I've tried to set lower baud rate - unfortunately I'm limited to 406800, because at lower values port can't keep up. But even when I configure receiver to send only one message (NAV-HPPOSLLH) and set baudrate to 38400 - problem still occurs.
2. When I just read messages from port - everything works excellent even on 921600 baud rate.
3. On MON-COMMS and MON-TXBUF all pendings = 0.
4. On u-blox forum they suggested me to turn on/off receiver that way. In u-center there is panel described as "Control GNSS" and in ZED-F9P interface decription it is stated as "Controlled GNSS stop/start"
   

CFG-RST:

BUT! :D

What I've noticed:

1. After I send message to start GNSS it replies with some "NMEA" like message started with $G, that the receiver is started. And I think that this message causes problem, which carries over to the rest of the messages. I only don't know how and why UBXReader is able to read several messages OK until it hangs up. Maybe here is some possibility that UBXReader would filter out such messages?

2. When I configure receiver to send also NMEA message (i.e. GNGGA) it solves problem! :D

Tommorow, or on Monday I will also try to send RTCM messages (to get RTK, centimeter level position) to the receiver every 1 second. I will let you know if there will be any problems then.

[semuadmin]

The UBXReader.read() method shouldn't have a problem with non-UBX (e.g. NMEA) data - provided the validate flag is set to the default 'False', it does effectively 'filter them out'.

I don't have an F9P device to hand but I've run your modified code through a standard-precision device (using NAV-POSLLH rather than NAV-HPPOSLLH) at 10Hz and 921600 baud (nominal) and it seems to run fine the first time, but glitches after a few seconds on any subsequent run. If, however, I explicitly set the navBbrMask in CFG-RST to b'\x0000' (rather than 'b\x00\x00', which is what UBXMessage.read() defaults it to) the problem goes away and I can run the code repeatedly without difficulty.

Suggest trying the following in the relevant sections in your code and see if it improves matters:
START GNSS - msg = UBXMessage(6, 4, SET, navBbrMask=b'\x0000', resetMode=9)
STOP GNSS - msg = UBXMessage(6, 4, SET, navBbrMask=b'\x0000', resetMode=8)

If this works consistently, I may need to make a minor tweak to ubxmessage.py.

This is the modified code that seems to work OK for me:

import time
import serial
from pyubx2 import UBXReader, UBXMessage, SET

PORT = 'COM13'
BAUD = 921600
TIMEOUT = 1
ITERATIONS = 10


print("Hello")

ser = serial.Serial(port=PORT, baudrate=BAUD, timeout=TIMEOUT)
print(f"Serial Port is open? -> {ser.is_open}")

if ser.is_open:
    x = 0
    while x < ITERATIONS:

        print(f"\n\nITERATION {x}\n\n")
        # START GNSS
        print("Sending CFG-RST GNSS Start!")
        msg = UBXMessage(6, 4, SET, navBbrMask=b'\x0000', resetMode=9)
        print(msg)
        ser.write(msg.serialize())
        print("NSS Started!")
    
        # READING UBX
        i = 0
        ubr = UBXReader(ser)
        while i < 300:
            print(f"Reading {i} :")
            (raw_data, parsed_data) = ubr.read()
            print(raw_data)
            print(parsed_data)
            if parsed_data.identity == 'NAV-POSLLH':
                print(f"\n\nLAT = {parsed_data.lat/10**7}, LON = {parsed_data.lon/10**7}")
            print("\n")
            i += 1
    
        # STOP GNSS
        print("Sending CFG-RST GNSS Stop!")
        msg1 = UBXMessage(6, 4, SET, navBbrMask=b'\x0000', resetMode=8)
        print(msg1)
        msg1 = msg1.serialize()
        ser.write(msg1)
        print("GNSS Stopped!")


        x += 1

    ser.close()
    print("Serial Port Closed!")

else:
    print("Problem with serial port")

[djdan7]

If I set navBbrMask it does not start nor stop receiver :/

PS. For what you do u-blox should send you a C099-F9P for free. Have you tried to contact them?

[semuadmin]

Well navBbrMask has to be set in code one way or another - it's a mandatory payload field. It's either defaulting to b'\x00\x00' (which is what UBXMessage defaults unspecified 2-byte bitmasks to), or you set it explicitly. Not sure why setting it explicitly would prevent the stop/start from working? According to the F9P data sheet, 'b\x0000' is the appropriate setting for 'hot start':

And, yes, I did try blagging some free devkit from u-blox but they weren't having it :-(

[djdan7]

I've set navBbrMask to b'\x0000' - it didn't work, so then I've copied all code, which you provided (and changed com port) - it also didn't work. What I mean saying didn't work:

1. If receiver was running (i.e. I started receiver via u-center) and then I run script - it read messages ok, but at the end it didn't stoped it
2. If receiver was stopped - it didn't start it, read 2 MON-RF messages (receiver send them no matter if it is stopped or not) and throws error:

if parsed_data.identity == 'NAV-POSLLH':
AttributeError: 'NoneType' object has no attribute 'identity'

I think that 'Controlled GNSS stop or start' =/= Hot start. When I stop GNSS for couple of minutes - it performs hot start, but when I stop it for i.e. 1 hour - it performs warm start, etc. So it not always performs hot start

What a shame for the u-blox. A decent reading library for ubx (like yours) in Python makes people from non-IT fields (like me) interested in their product. I think Python is the only simple language for people outside the IT industry. I ordered it about 3 months ago and the package arrived after a week.

[semuadmin]

OK leave it with me for now - I'll need to run some comparative tests against u-center to see exactly how u-center is populating the navBbrMask field in the CFG-RST GNSS stop/start message.

[djdan7]

Ok, thank you so much for your effort!

[semuadmin]

OK so I've just run a comparative test against u-center and this is the binary output from u-center showing a CFG-RST stop (highlighted in yellow) followed by a CFG-RST start (highlighted in green), followed by a resumption in NAV messages:

In both cases, the navBbrMask field (bytes 6 & 7 of the message; bytes 1 & 2 of the payload) is set to hex 00 00 (which is exactly what pyubx2 should be setting it to by default). In both cases, the UBX CFG-RST message is 'acknowledged' by one or more NMEA TXT messages (a confirmation and - after a stop - an antenna status), as you've observed.

So I need to understand why we appear to be getting different results from the device when sending apparently identical messages from u-Center and pyubx2....leave it with me.

[semuadmin]

OK I believe this is sorted now with a fix to UBXReader which I've just uploaded in v1.0.3. Basically, you called it in your original issue report but I managed to lead us down a wild goose chase with CFG-RST message formats - doh! The CFG-RST message handling was fine - the problem was in the parsing of the subsequent NMEA TXT responses. Surprised this hasn't cropped up before.

I've rerun my modified version of your test code (with navBbrMask left at the default b'\x00\x00') and it seems to be working fine now. GNSS is successfully stopped at the end and I'm seeing no further UBX messages until I explicitly start it again. Have a go yourself and let me know if it works for you.

[djdan7]

Thank you!
After updating to v1.0.3 everything works GREAT!!! :D

If I can spam a little more here, I'd have a question. What is the appropriate method to also read the NMEA message once every 10 seconds? I need to read a GNGGA message, so then I can send it to the NTRIP client and download the appropriate RTCM corrections. At this point, I wrote something like this, but I predict that it could be done a little better:

import datetime

import pyubx2.exceptions as ube
import serial
from pyubx2 import UBXReader, UBXMessage, SET


# START GNSS
def start_gnss(ser):
    if ser.is_open:
        msg = UBXMessage(6, 4, SET, resetMode=9)
        ser.write(msg.serialize())
        print(msg)
        print(datetime.datetime.now())
        print("STARTED!")
    else:
        print("PORT CLOSED")


# READING UBX
def read_ubx(ser, ubr):
    if ser.is_open:
        i = 0
        while i < 1:
            try:
                (raw_data, parsed_data) = ubr.read()
                print(raw_data)
                print(parsed_data)
                print(datetime.datetime.now())
                if parsed_data.identity == 'NAV-HPPOSLLH':
                    print("LON = ", (parsed_data.lon + parsed_data.lonHp / 100) * 10 ** (-7))
                    print("LAT = ", (parsed_data.lat + parsed_data.latHp / 100) * 10 ** (-7))
                print("\n")
                i += 1
            except ube.UBXStreamError:
                try:
                    print("EXCEPT NMEA")
                    gga = "$" + str((ser.readline()).decode())
                    print(gga)
                except:
                    print("EXCEPT NMEA EXCEPT")
                    ubr = UBXReader(ser, True)
                    ser.flushInput()
            except:
                print("EXCEPT")
                ubr = UBXReader(ser, True)
                ser.flushInput()
    else:
        print("PORT CLOSED")


# STOP GNSS
def stop_gnss(ser):
    if ser.is_open:
        msg = UBXMessage(6, 4, SET, resetMode=8)
        ser.write(msg.serialize())
        print(msg)
        print(datetime.datetime.now())
        print("STOPPED!")
    else:
        print("PORT CLOSED")


print("Hello")

ser = serial.Serial(port="COM9", baudrate=921600, timeout=1)
print("Port is open? -> ", ser.is_open)

start_gnss(ser)

ubr = UBXReader(ser, True)
for i in range(300):
    print(i)
    read_ubx(ser, ubr)

stop_gnss(ser)

ser.close()

It works fine (I have everything I need), but it read NMEA messages at 10Hz unnecessarily. Would it be better to send a message every 10 seconds to configure the receiver to send GNGGA, then read this message and send the configuration message again so that it stops sending GNGGA? Do I need to use "UBX-CFG-VALSET"? I'm not 100% sure how to do it exactly. Thank you in advance! :D

[semuadmin]

OK glad it works and sorry for the wild goose chase! You've also prompted me to make a corresponding change to the partner GUI application PyGPSClient.

In terms of your use case, few observations that might help:

1. Might be more robust to have the reader running as a daemon thread and send any CFG command messages via a separate 'command' thread - have a look at the '\examples\ubxstreamer.py' example.
2. For Generation 9+ devices like yours, you can use a CFG-VALSET message to set the CFG-RATE-MEAS parameter (keyid 0x30210001) to the required interval in ms (1000 = 1Hz, 100 = 10Hz, etc). As you know, this affects the frequency at which the GNSS takes position measurements.
3. If, on the other hand, you simply want to receive certain UBX or NMEA message types more or less frequently, you can use CFG-VALSET to set the corresponding CFG-MSGOUT parameter e.g. to receive GNGGA messages on the USB port every 10 seconds at a GNSS measurement rate of 10Hz, you could use CFG-VALSET to set CFG-MSGOUT-NMEA-ID-GGA-USB (keyID 0x209100bd) to a value of 100, meaning it will send one GGA message out on every 100th position measurement. It won't necessarily be exactly every 10 seconds but I think it gives you what you're after.

You might also like to check out PYGPSClient. It's not intended to be a u-center replacement but, unlike u-center, it runs on any platform and it provides a 'ubxpresets' facility allowing users to configure frequently-used message sequences (e.g. in your case a GNSS stop/start or a change in message rate). It also has a somewhat less daunting UI (IMHO!).

Hope this helps. Thanks for picking this up. Are you OK for me to close this issue now?

[djdan7]

Thank you soooo much! :D

Of course, you can close this issue :D

[semuadmin]

Cheers.

PS if you're liking the library, it would help me if you could 'Star' it in Github :-)

[Nerolf05]

Hi, firstly thank you very much for the great package

regarding the issue of the stuck UBXReader I would also like to ask a question:

At first be aware that the ubx-cfg-rst should not be expected to be acknowledged by new receivers, which will result in an infinity reading loop as long as no other ubx-message is written into the buffer by the Receiver:

(https://www.u-blox.com/en/docs/UBX-18010854)

But I encounter the same problem with a F9P, when sending a ubx-cfg-valset message, which is expected to be acknowledged (by ubx-ack-ack or at least ubx-ack-nak), and directly starting to read with the UBXReader.read()- method runs into an infinity loop, as no ubx-message is read.

The problem can be circumvented by sleeping for more than 0.1 seconds after writing to the device, but I think this is not so nice, and can be avoided by the proposed usage of a reading-thread.
Hereby I would like to ask, how to pass the read UBXMessages in a memory-safe way from the read-thread into the main-thread?
Do you use a Queue?

Best regards

[semuadmin]

Hi - so firstly, yes I am aware that the CFG-RST message won't receive an acknowledgement. Indeed, it may result in a serial port 'ClearCommError' requiring a restart of the device port (I mention this explicitly in the CFG-RST example in my sample ubxpresets file). If you're implementing an application that sends CFG-RST messages frequently you could choose to handle this in a suitable try-except loop.

On your second point, apologies but I'm not sure I understand the question. What particular CFG-VALSET message are you sending? Are you using the PyGPSClient UBXConfig facility or do you have some other sample code? In my examples the serial read thread is in effect an 'infinity loop' which is only terminated when you close the serial port, so I'm unclear in what sense this is not expected behaviour?

[Nerolf05]

Thanks for your fast reply
Sorry for the confusing second point:

Basically I would like to configure+read/poll ublox-gnss Receivers with a python script:

class UbloxReceiver:
...

def _ubx_communicate(msg: UBXMessage, expect_ack: bool, expect_msg: bool, timeout: float) -> typing.Union[UBXMessage, bool]:
    ...
    if msg.msgmode != GET:
        self._serial_object.write(msg.serialize())
        time.sleep(0.5)  # workaround
    while check_timeout is False:
        try:
            (_, parsed_data) = self._ubxreader.read():
            ....
             if parsed_data.identity == msg.identity and expect_msg:
                   return parsed_data
            if 
        except ...
    return False

Right now I do not use a reading-thread, but I would like to use one, as it clearly seems to be more stable, than my current approach. My Question would be:
How to pass a UBXMessage-object from the reading Thread into my function _ubx_communication() instead of the direct use of self._ubxreader.read().
In detail, I would like to know, how to pipe the parsed_data into another thread, instead of writing into the console.

[semuadmin]

Hi Nerolf05 - sorry for the delay in getting back to you. I think this is more a general Python thread-handling question than a specific pyubx2 issue, but if I understand your requirement correctly, you want to send commands to the device while at the same time monitoring the output. That's basically exactly what the '\examples\ubxstreamer.py' example does, so I recommend you take a look at that. Alternatively, if you want a more sophisticated example you could take a look at the implementation in the PyGPSClient GUI application (specifically the 'serial_handler.py' and 'ubx_handler.py' modules). I'm closing this issue thread down now, but I hope this helps, and thanks again for your interest in the pyubx2 library.