Not receiving GPS position over tcp and pty on socktap

[daniel0720]

Hello everyone,
Since I don’t have an actual gps receiving device, I use gpsfake to generate simulated gps data. The gps log file used is nmea format data. See attachment 1 for details.
nmea.log

The version of gpsd is 3.22, as shown in the figure below
The sequence of operations is to start gpsfake and then run socktap.
First run gpsfake in pty mode, the command is sudo gpsfake -D3 -S nmea.log. Then run the socktap command. At this time, the output of gpsfake is as shown below.

socktap also reported an error

I use the gpsmon command to view gps information, the output is as follows

Then I saw in the issue that it was said to use tcp mode to send GPS data, so I used tcp instead. When using the tcp protocol, the socktap program no longer reports an error and can send BTP-B data packets, but the latitude and longitude information in them are all 0. gpsfake output is as follows

Use gpsmon to view gps information and get gps data.

I don’t understand why socktap cannot obtain the gps data sent by gpsfake.
Can anyone help me?
Thanks a lot.

[daniel0720]

By the way, the system is Ubuntu22.04

[daniel0720]

It seems that the program stucks in this. gps-data.devices.ndevices=1

[riebl]

Hi @daniel0720, so the GpsPositionProvider works as expected if you have removed the gps_data.devices.ndevices > 0 condition?

[daniel0720]

Hi @riebl , I modified the function GpsPositionProvider::fetch_position_fix() as follows. In this way, I can get the BTP-B packet with gps longitude and latitude. The maintainer of gpsd says there are some bugs in the code of gps_position_provider.cpp and he is happy to help to fix it.

void GpsPositionProvider::fetch_position_fix() {
    int gps_read_rc = 0;
    do {
        gps_read_rc = gpsd_read(gps_data);
        if (gps_read_rc < 0) {
            throw GpsPositioningException(errno);
        }

        if (MODE_SET != (MODE_SET & gps_data.set)) {
            // did not even get mode, nothing to see here
            continue;
        }

        //if (gpsd_has_useful_fix(gps_data)) {
        using namespace vanetza::units;
        static const TrueNorth north = TrueNorth::from_value(0.0);

        fetched_position_fix.timestamp = convert_gps_time(gps_data.fix.time);
        fetched_position_fix.latitude = gps_data.fix.latitude * degree;
        fetched_position_fix.longitude = gps_data.fix.longitude * degree;
        fetched_position_fix.speed.assign(gps_data.fix.speed * si::meter_per_second,
                                          gps_data.fix.eps * si::meter_per_second);
        fetched_position_fix.course.assign(north + gps_data.fix.track * degree, north + gps_data.fix.epd * degree);
        if (!std::isnan(gps_data.fix.epx) && !std::isnan(gps_data.fix.epy)) {
            if (gps_data.fix.epx > gps_data.fix.epy) {
                fetched_position_fix.confidence.semi_minor = gps_data.fix.epy * si::meter;
                fetched_position_fix.confidence.semi_major = gps_data.fix.epx * si::meter;
                fetched_position_fix.confidence.orientation = north + 90.0 * degree;
            } else {
                fetched_position_fix.confidence.semi_minor = gps_data.fix.epx * si::meter;
                fetched_position_fix.confidence.semi_major = gps_data.fix.epy * si::meter;
                fetched_position_fix.confidence.orientation = north;
            }
        } else {
            fetched_position_fix.confidence = vanetza::PositionConfidence();
        }
        if (gps_data.fix.mode == MODE_3D) {
            fetched_position_fix.altitude = vanetza::ConfidentQuantity<vanetza::units::Length>{
                    gpsd_get_altitude(gps_data) * si::meter, gps_data.fix.epv * si::meter};
        } else {
            fetched_position_fix.altitude = boost::none;
        }
        //}
    } while (gps_read_rc > 0 && gps_data.devices.ndevices > 0);
}

[garyemiller]

Hi! Gary E. Miller here, a maintainer for gpsd. Several things to note here:

The version of gpsd is 3.22

Version 3.25 is current.

sudo gpsfake -D3 -S nmea.log.

gpsfake does not require sudo, or root. As the man page says:

"gpsfake does not require root privileges, but will run fine as root. It can be run concurrently with a production gpsd instance without causing problems, as long as you use the -P option. Running under sudo will cause minor loss of functionality."

The code in gps_position_provider.cpp has several bugs. The example1.c program for gpsd shows how to properly check for a valid fix:

https://gpsd.io/gpsd-client-example-code.html

One major bug is that assumes that fix.status tells you anything about whether the location fix is present or valid.

[riebl]

Thanks @garyemiller for input. The upstream sources of Vanetza no longer use fix.status but fix.mode for some time already, so please pull the latest sources @daniel0720.

[garyemiller]

I'm unfamiliar with this project. Where are the upstream sources? Did they fix the other bugs?

[riebl]

The upstream sources are just here in the master branch. Because of this issue ticket, I have revised the GpsPositionProvider in a feature branch: https://github.com/riebl/vanetza/blob/socktap_gpsd/tools/socktap/gps_position_provider.cpp
Among others, I have replaced std::isnan by std::isfinite and I check MODE_SET and TIME_SET before reading the corresponding fields. Do you have further remarks @garyemiller? I am happy to consider them and merge this feature branch then into master.

[garyemiller]

I see things that I have complained about still unfixed.

Did you fix this weirdness?

     if (gps_data.fix.epx > gps_data.fix.epy) {
                fetched_position_fix.confidence.semi_minor = gps_data.fix.epy * si::meter;
                fetched_position_fix.confidence.semi_major = gps_data.fix.epx * si::meter;
                fetched_position_fix.confidence.orientation = north + 90.0 * degree;
            } else {
                fetched_position_fix.confidence.semi_minor = gps_data.fix.epx * si::meter;
                fetched_position_fix.confidence.semi_major = gps_data.fix.epy * si::meter;
                fetched_position_fix.confidence.orientation = north;
            }

epx and epy have nothing to do with the semi_minor and semi_major axis!!

And this:

void GpsPositionProvider::fetch_position_fix()
{
    int gps_read_rc = 0;
    if (uses_shm_ || gps_waiting(&gps_data_, 0)) 

I recommend against using uses_shm. The SHM interface is very unstable and this project can't keep up with the much more stable JSON interface.

Only surveyors use altHAE. I can get pilots killed:

{
#if GPSD_API_MAJOR_VERSION > 8
    return data.fix.altHAE;

[garyemiller]

I see things that I have complained about still unfixed.

Did you fix this weirdness?

     if (gps_data.fix.epx > gps_data.fix.epy) {
                fetched_position_fix.confidence.semi_minor = gps_data.fix.epy * si::meter;
                fetched_position_fix.confidence.semi_major = gps_data.fix.epx * si::meter;
                fetched_position_fix.confidence.orientation = north + 90.0 * degree;
            } else {
                fetched_position_fix.confidence.semi_minor = gps_data.fix.epx * si::meter;
                fetched_position_fix.confidence.semi_major = gps_data.fix.epy * si::meter;
                fetched_position_fix.confidence.orientation = north;
            }

epx and epy have nothing to do with the semi_minor and semi_major axis!!

And this:

void GpsPositionProvider::fetch_position_fix()
{
    int gps_read_rc = 0;
    if (uses_shm_ || gps_waiting(&gps_data_, 0)) 

I recommend against using uses_shm. The SHM interface is very unstable and this project can't keep up with the much more stable JSON interface.

Only surveyors use altHAE. I can get pilots killed:

{
#if GPSD_API_MAJOR_VERSION > 8
    return data.fix.altHAE;

[garyemiller]

This is wrong:

        gpsd_get_altitude(gps_data) * si::meter, gps_data.fix.epv * si::meter };

epv units is already in meters/second. What is the si:meter multiplying doing??

[riebl]

Thanks for your remarks @garyemiller!

I see things that I have complained about still unfixed.

Where did you note down your complaints? I must have missed these because they are not in this issue ticket.

epx and epy have nothing to do with the semi_minor and semi_major axis!!

I have just discovered GST_SET and the accompanying gst_t structure containing the error ellipse attributes. I will change the code accordingly.

Is it good practice to constantly merge gps_data_t readings with gps_merge_fix so GST data remains available?

Only surveyors use altHAE. I can get pilots killed:

Vanetza implements ETSI ITS standards, where an altitude of zero refers to "referenceEllipsoidSurface" in the WGS84 coordinate system; thus, I think altHAE is the correct date. No pilots involved, only drivers and riders ;-)

epv units is already in meters/second. What is the si: meter multiplying doing??

Multiplying with si::meter converts a plain (double) value to a quantity carrying the value with a unit. No need to worry.
The gps.h header on my system says double epv; // Vertical position uncertainty, meters.
I recognize that libgpsd_core.c describes the calculation of epv as "dodgy", so I guess gst_t.alt_err_deviation should be preferred. Is my assumption correct?

[garyemiller]

Where did you note down your complaints? I must have missed these because they are not in this issue ticket.

In the gpsd issue. but best to just move it all here.

epx and epy have nothing to do with the semi_minor and semi_major axis!!

I have just discovered GST_SET and the accompanying gst_t structure containing the error ellipse attributes. I will change the code accordingly.

Very few GNSS receivers output GST_SET. You can ignore that for now.

Is it good practice to constantly merge gps_data_t readings with gps_merge_fix so GST data remains available?

Nope. Grab what you need as it flies by. gps_data_t has a huge number of things you do not care about.

Only surveyors use altHAE. I can get pilots killed:

Vanetza implements ETSI ITS standards, where an altitude of zero refers to "referenceEllipsoidSurface" in the WGS84 coordinate system; thus, I think altHAE is the correct date. No pilots involved, only drivers and riders ;-)

If that's the spec, then fine. But altHAE can differ from altMSL by up to 200m. People expect altMSL.
OTOH, few GNSS receivers calcualte altMSL properly.

epv units is already in meters/second. What is the si: meter multiplying doing??

Multiplying with si::meter converts a plain (double) value to a quantity carrying the value with a unit.

Odd, but if that is how it works on your end. I also take back what I said. epv is ep of vertical, so meters is correct. I always misread that as velocity, which would be m/s.

No need to worry. The gps.h header on my system says double epv; // Vertical position uncertainty, meters. I recognize that libgpsd_core.c describes the calculation of epv as "dodgy",

Yes. As good as it gets, without an IMU, but not good.

so I guess gst_t.alt_err_deviation should be preferred. Is my assumption correct?

No, not, not at all. That is for pseudo ranges! There is one psuedorange per sat in use, and it is the guesstimated distance from the antenna to that sat. Only after some really ugly math does that become part of the ep's.

Stick to the TPV sentences.

[riebl]

Stick to the TPV sentences.

I would love to stick with epx, epy, and epv. Ultimately, we need to conform to ETSI's ReferencePositionWithConfidence.
However, I am puzzled now about the meaning of these epX values. My reading of https://gpsd.gitlab.io/gpsd/gpsd.html#_accuracy is that these fields are the predicted accuracies with 95% confidence. I call them "predicted" because they depend only on DOP and UERE constants (labeled with 95% confidence in the code), i.e. no pseudorange measurements are considered.

ETSI's AltitudeConfidence shall express the "estimated absolute accuracy". The same wording "estimated accuracy" is used for the PositionConfidenceEllipse. Since the ellipse is only given by GST, my gut feeling is that GST error estimates should be used for both, altitude and position confidence.
At least it sounds utterly dangerous to mix GST data for the position confidence ellipse and epv for the altitude confidence in the very same ReferencePositionWithConfidence structure.

If epv is used, e.g. as a fallback when GST is not provided, then the ellipse should be filled from a similar source too. That's the current "weirdness" you have noted. Assuming that epx is 2*sigma_x and epy is 2*sigma_y (see above), we could derive an axis-aligned 95% confidence ellipse with semi-axis lengths of 0.5*epx*sqrt(5.991) and 0.5*epy*sqrt(5.991) based on a Chi-Square distribution. As a fallback, I would also accept a 95% confidence circle as degraded ellipse based on eph.

I guess it boils down to the question if epx and its siblings have any useful meaning.

[garyemiller]

https://forge.etsi.org/rep/ITS/asn1/cdd_ts102894_2/-/blob/v2.1.1/docs/ETSI-ITS-CDD.md#ReferencePositionWithConfidence

That anchor does not exist in that document.

However, I am puzzled now about the meaning of these epX values.

You, and everyone else that ever looks at them.

My reading of https://gpsd.gitlab.io/gpsd/gpsd.html#_accuracy is that these fields are the predicted accuracies with 95% confidence.

You need to read more carefully. Here is the ONLY place that document mentions epx, epy and epv:

Generation of position error estimates (eph, epv, epd, eps, epc) from the incomplete data handed back by GPS reporting protocols involves both a lot of mathematical black art and fragile device-dependent assumptions. This code has been " in the past and problems may still lurk there.

Key phrases:

"fragile device-dependent "

"bug-prone"

"problems may still lurk there."

That is a nice way of saying the numbers are all hand waving. Every receiver that bothers to calculate them does so differently. There are no standards, or even any documentation, on how each receiver calculates them. You can only use these as a rough "figure of merit".

I call them "predicted" because they depend only on DOP and UERE constants (labeled with 95% confidence in the code), i.e. no pseudorange measurements are considered.

How do you think DOPs are calculated? EVERYTHING starts with the psuedoranges.

ETSI's AltitudeConfidence shall express the "estimated absolute accuracy".

Actually, it is MUCH more specific than that. So specific that I know of know receiver on the market that can satisfy their requirements.

Since the ellipse is only given by GST, my gut feeling is that GST error estimates should be used for both, altitude and position confidence.

I would love to get the GST error estimates. But so far only a very few u-blox receivers provide them. If you have that model receiver, then you will get the GST automagically. Otherwise, you are totally out of luck.

Assuming that epx is 2sigma_x and epy is 2sigma_y

Once again you can not "Assume". The spec you want does not want you to assume, it wants you to know. Sadly, very few receivers tell you how they calcualate epx, epy and epv. Of the few that do, they use 50%, 1 sigma, 2 sigma, 3 sigma, and other measures. That is no way to get 2 sigma unless you have a receiver that tells you it is giving you 2 sigma. And even those lie.

we could derive an axis-aligned 95% confidence ellipse with semi-axis lengths of 0.5epxsqrt(5.991) and 0.5epysqrt(5.991) based on a Chi-Square distribution.

One of the things you quickly learn with GNSS is that the simple academic distributions you were taught are useless when doing geodesy.

Stop piling assumptions on assumptions!

I guess it boils down to the question if epx and its siblings have any useful meaning.

Yes, they do. For any given receiver and antenna combination they are very useful as a "figure of merit". Beyond that, they tell you very little.

Now, step way, way back, and think about this from a very high level. Almost all that gpsd does is pass on the data that your GNSS receiver gives it. When the data is sparse, like you have altHAE, but not altMSL, then gpsd will try to compute the missing value. But even then the "accuracy" depends on the "accuracy" of the data (altHAE) from the receiver.

gpsd can not pull data out of thin air. gpsd, and it clients, can only guess the quality of the data by reading the (false) data sheets, and running tests.

Only a handful of receivers say they output a 95% confidence epx, epy, and epv. Most of them are not telling the truth. If you really need 95%, you need to hand select each receiver and antenna, then test it for a long time.

Even "95%" is a really sloppy measure. Is that "95%"
a day? Under normal conditions? Or does that "95%' ove an entire sun cycle including the time of solar storms?

It is not possible to actually comply with your "standard". You have to decide what is "good enough".