Assisting

[larsnaesbye]

Hi, I might want to contribute to the development of rinex, as my day to day work is in need of better tools than tecq, Anubis, gfzrnx etc. I have experience with RINEX, and though fairly new to Rust, I'm already fascinated by the tooling and community.

Are there some smaller tasks you'd like to be worked on first, like contributing test data, hardening or fixing the epochs?

[gwbres]

Hello @larsnaesbye, welcome onboard!

At the moment it is fair to say that the parser is the most stable part of the project.
Navigation, Observation, Crinex (compressed Obs) and Meteo records are very stable (up to V3). I just marked Antex and Clocks as stable in the last few days. You can refer to the entire /test_resources database, which is used for automated test purposes (for information, there are about 40 tests run automatically at the moment, mostly about the parser).

The rinex directory is a rust library, which everyone can include and take advantage of, the official documentation can be found here. It is shipped with a couple of examples, you can browse their codes in rinex/examples directory, I should get back to enhance them and provide more examples (clocks, antex..).
For example, you can run the observation example with:

cargo run --example observation

I am most focused on the parser (file reader) at the moment as opposed to

• the command line tools: which you are probably more interested in
• and the file writer (file production) which you might be interested in

I am currently concluding support for IonosphereMaps and Troposphere data, then I will get back to the command line tool.

Regarding the command line tools, rinex-cli is an application, written in rust. That means it builds easily with "cargo" but it is supposed to be run in a shell. It uses the main library (but hidden to the user), so the user can easily take advantage of it. The command line application is somewhat capable of analyzing OBS/NAV/MET. It is not capable of plotting at the moment, I will try to provide that in a near future. There is a lot that can still be achieved with this tool. If you want to give it a try, you can do something like this:

cd rinex-cli
cargo run -- --help 
# print available obscodes in this RINEX
cargo run -- -fp ../test_resources/OBS/V3/DUTH0630.22O --obscodes --pretty
# extract data for Sv "G01" but only for epochs with an `ok` flag attached to them:
cargo run -- -fp ../test_resources/OBS/V3/DUTH0630.22O -e --sv G01 --pretty

As for the teqc-like operations, which you are interested in, there are three things we can say:

• the action takes place at the Rinex structure level, see split_record_at_epoch() and neighboring methods,
  like merge().
• they are not too complex, but very verbose.. because the core differs for every record type (hidden to user).
  I barely spent any time on these methods at the moment. I would say, the best thing to do is to focus on the merge() ops, and test it properly in rinex/tests/merge.rs. I am also looking forward to providing a down_sample() method.
• once we stabilize such methods, it can easily be exposed in the command line tool to create a probably similar (and possibly more advanced) teqc operation

Overall, and I will conclude with this:

• I started using RINEX this year and I still have doubts / questions about things.
  I also struggle to find any test data, I don't have a single V4 for instance.
  If we can discuss things in the future, just that would be already valuable help. My job involves timing and related measurements (hence the ClockData support), I am looking forward to providing proper Time System conversions and related calculations
• the record manipulation is the most complex thing in this project. First of all because the record type depends on the type of file, so they are all different. Then, because it involves iterator. Iteration is very efficient and powerful, it provides generic filter and sorting methods (see the example codes), but it is also very verbose and hard to grasp in my opinion.
• for example, OBS/NAV/MET are all iterated on an epoch basis, a merge operation will obviously involve epoch iterations
• So far I have used 3 structures to declare Data Arrays:
  • Vec : standard Vector indexed by an integer, they are not sorted! order is unpredictable
  • Hash: Vector but indexed by an object (which must implement == comparison). I used Space vehicule indexing a lot, which sorts data efficiently
  • BTreeMap : same as Hashmap but data is ordered! (must implement Ord). I use BTreeMaps to have the record chronologically sorted on an epoch basis

Now that is already a lot of information. Let me know if you want to know more.

[larsnaesbye]

Hello @gwbres,

Thank you for the very descriptive welcome! There is absolutely everything I would like to know in there for now.

You are right, the file production and command-line tools are what I ultimately would like to use for daily operations. I oversee a system that produces RINEX files from raw receiver data and distributes them freely for public consumption.

In short, the tasks are:

• Convert raw binary from receiver to RINEX - obviously not in scope for this project
• Pack and unpack gzip
• Compress and decompress Hatanaka
• Rewrite RINEX headers
• Splice 1-hour files into 24-hour files
• Decimate 1-sec files into 30-sec files
• Gap analysis
• QC on the 30-sec files

For this we use a combination of gfzrnx, bnc, Anubis and CRX2RNX/RNX2CRX (the one by Hatanaka). If these tasks could in the future be done using the same framework, it would make everything easier and more transparent.

I think I can pretty easily get some RINEX v4 files, since the RAW to RINEX software supports it. Let me get back to you on that.

After that I might look into the rinex-cli, if that's OK with you.

[gwbres]

Hello @larsnaesbye ,

I oversee a system that produces RINEX files from raw receiver data and distributes them freely for public consumption

I initiated a "ublox-rinex" application, it is also contained in this folder, and it serves that role as long as you intend to use "ublox" receivers. It basically combines the "ublox" crate (very powerful! I use it at work on a daily basis), and the "rinex" crate (which already knows the most common RINEXes), that could be of interest for you. I barely spent any time on this application. We need to conclude the Rinex::to_file() method to make this application happen. Which is not too painful and should not take long. After that, we can combine the best of both worlds. If you intend to use different receivers, obviously that will not help.

Convert raw binary from receiver to RINEX - obviously not in scope for this project

The rinex-ublox app does that from a ublox receiver. The official ublox crate lacks some Ublox frames, therefore I have the application point to my fork (timing, and other ubx frames)

Pack and unpack gzip

I just added the hability to parse .gz files directly. I am improving the BufferedReader (most efficient way to browse the file) in the sense it is already capable to uncompress .gzip in a hidden fashion and soon Hatanaka too (makes things much easier).
If we start tackling the to_file() side of things, we would need the mirror operations, BufferedWritter for efficient file production and smooth operations.

Compress and decompress Hatanaka

Supported natively, will be managed by the BufferedReader directly in my next commits.
Note I have not completed the Hatanaka compression (mirror operation) in the Hatanaka core structures! needs to be finalized to have an efficient to_file()

Rewrite RINEX headers

Rinex::to_file() is partial, it needs to be improved, support a buffered stream efficient approach, support all known attributes and record types..

Splice 1-hour files into 24-hour files

I have not initiate the Rinex::splice operation, because I do not know what splicing means

Decimate 1-sec files into 30-sec files

Decimate/down sampling is under development but is closer to stable than all other topics.
See Rinex::resample(std::time::Duration) to rework a sampling interval. Or Rinex::Decimate(N) to reduce the epoch rate by N

Gap analysis

See Rinex::dead_times() for gap analysis, also Rinex::anomalies() for epoch flags analysis (flags only truly relevant in OBS files)

[larsnaesbye]

I initiated a "ublox-rinex" application, it is also contained in this folder, and it serves that role as long as you intend to use "ublox" receivers. It basically combines the "ublox" crate (very powerful! I use it at work on a daily basis), and the "rinex" crate (which already knows the most common RINEXes), that could be of interest for you. I barely spent any time on this application. We need to conclude the Rinex::to_file() method to make this application happen. Which is not too painful and should not take long. After that, we can combine the best of both worlds. If you intend to use different receivers, obviously that will not help.

Yeah, we use and have used other types of receiver (Leica, Trimble, Septentrio...), not u-blox.

I have not initiate the Rinex::splice operation, because I do not know what splicing means

I think splice and merge is the same thing - combining two files from the same receiver to get one continuous. A bit like they describe on the tecq mailing list here: https://postal.unavco.org/pipermail/teqc/2016/002144.html

I have submitted a pull request with a RINEX 4 file (Hatanaka 3 compressed). I hope it can be of use for testing :-)

[gwbres]

Hello @larsnaesbye

I validated some high level record operations, I started integrating them into rinx-cli, which will reduce the code quantity and make things much smoother / efficient. In rinex-cli, by using the mut() declination of filters, we can stack them together and create interesting operations.

Here is my notation:
✔️ coded, tested, integrated to autotests
🎇 should work

• decimate_by_interval(duration) ✔️
  At the moment, it is not a feasible operation if the header section did not come with an INTERVAL field.
  Which is a bit silly.. A method to determine the most plausible epoch interval (average, histogram peak..) should exist

• decimate_by_interval_mut(duration) ✔️
  same thing, but decimates in place

• decimate_by_ratio(integer) ✔️
  same thing but works as a modulo operation, not from epoch intervals

• decimate_by_ratio_mut(integer) ✔️
  same thing but decimates in place

• lli_filter (mask) : 🎇
  applies AND mask on LLI data in this observation rinex.
  Has no effect on Non Observation Data.

• lli_filter_mut(mask) : 🎇
  same thing, but works in place

• lock_loss_epochs: 🎇
  returns list of epochs where lock was declared lost.

• lock_loss_filter: 🎇
  cleans up epochs (in place) returned in previous method

• epoch_ok_filter_mut() 🎇 but i'm confident

• epoch_ok_filter() 🎇 but i'm confident

• data_gap 🎇
  returns list of data gaps, meaning epochs where |e(k)-e(k-1)| > interval

• constellation_filter_mut 🎇

• space_vehicule_filter_mut 🎇

• average_epoch_duration 🎇

[larsnaesbye]

This is really good. I think I will soon try to run a process flow and see if I can do the same with rinex-cli as I use 4-5 tools for today.

[gwbres]

@larsnaesbye

I added a note to the documentation on how to browse the different record types

Here are a few commands I can execute on my side, they seem to produce the expected results.
File generation is about perfect for meteo files, about 90% concluded for OBS, and just started for NAV.
I'll inquire the next PR tomorrow or in the next few days.
Once rinex-cli is stabilized, I will unlock an option to plot the data (GUI), not just print it, for a better user experience

# --decim-ratio : reduce information quantity
rinex-cli -- -f test_resources/OBS/V2/aopr0010.17o -o --decim-ratio 4 --pretty

# --decim-interval : provide a duration and reduce information efficiently
cargo run -- -f ../test_resources/OBS/V2/zegv0010.21o \ 
          --ssi 8 # excelent signal quality \
                 --decim-interval 00:01:00 # new option \
                         --pretty

# focus on GPS vehicules
cargo run -- -f ../test_resources/OBS/V2/zegv0010.21o \
             --constellation GPS \
                       --decim-interval 00:02:00 --pretty
                     
                     
# focus on G07
cargo run -- -f ../test_resources/OBS/V2/zegv0010.21o --sv G07 --decim-interval 00:02:00  --pretty

# split at epoch
# split operation, with an epoch specified, 
# we disregard whether this file is merged or not, specified epoch must exist
cargo run -- -f ../test_resources/OBS/V2/zegv0010.21o --decim-interval 00:01:00 --split "2021-01-01 00:09:00"
# note the "quotes ‘’, needed for the whitespaces

# this produces the couple (split1.txt, split2.txt). 
# There you see what I just pushed, with the first few to_file() methods made avaialble.
# There is no mean to properly control the names to be generated at the moment 

[larsnaesbye]

Looking good. I noticed that the v4 NAV file does make the tests fail. I might look into this too.

[gwbres]

Hello

@larsnaesbye
you might want to take a look at the methods I pushed today,

• space_vehicule_clocks_offset()
• space_vehicule_clocks_drift()
• pseudo ranges to geometric distances
  I'm not sure this one has an application ?
  we can obtain the space vehicules clock offsets from NAV frames,
  we might have the local clock offset on an epoch basis for this OBS,
  we can convert PR to real distance
• iono_free_pseudo_ranges()
  if we have observations from two separate carrier, on an epoch and vehicule basis
• iono_free_carrier_phases()
  if we have observations from two separate carrier, on an epoch and vehicule basis

[larsnaesbye]

That's really cool that you implemented this! I am sure many organisations have a need to set the headers to indicate where data comes from.

[gwbres]

Hello @larsnaesbye

by curiosity, would enhanced or improved performances, as opposed to supposedly less performant solutions (like interpreted python for instance), be something interesting to you? or at least make this library more valuable than other solutions

at the moment, the main interest I see in this library is the ability to process any kind of rinex (well.. almost, we'll get there), and span all revisions. I guess few other tools have this ability. The python lib I know is limited to OBS and NAV (not even sure) and probably has some revision limitations. Also I'm pretty sure very few support V4 like we do. But people did not take the V4 gap yet, I assume because of the previous point.

I hear often that Rust code is supposedly "more performant". It's compiled so we should out perform python libraries. But I also hear that it can achieve blazing performances. I know some portions of this lib can seriously be improved, i'm thinking about the OBS and NAV reading part. The compression part is quite optimized at the moment I would say. Anyway, this would come in a second phase where all major topics were completed first. I'll probably inquire some means to evaluate parsing/compressing performances at some point.

see you

[larsnaesbye]

Hmmm, I wonder which tools would be good for finetuning Rust performance.

[gwbres]

Hmmm, I wonder which tools would be good for finetuning Rust performance.

I looked into this a little bit, there is a benchmark infrastructure that can be added on top of builtin tests.
It is invoked in similar fashion, and has lots of objects and features to evaluate performances.
The only thing is, benchmarks can only run against the unstable rust compiler to this day, so it has to be isolated from the rest of the library.
BTW, in case you don't use it: I highly recommend using rustup to manage your Rust environments - its like a python manager.
I personally cross-compile rust code a lot, and it is very easy to do so with this tool.
I just ran a quick test of this whole benchmark thing, the test parsed the gzip compressed NAV4 you recently provided, you gets basic results like these:

rustup default nightly # set unstable rustc as default
rustup run nightly cargo bench # invoke "cargo bench" by temporarily switching to unstable compiler (very convenient)
bench:  19,910,022 ns/iter (+/- 835,513)  # 20 loops
bench: 103,932,756 ns/iter (+/- 3,091,624) # 100 loops, 

I'll probably add a benchmark infrastructure because it won't require much effort.
Results don't seem very meaningful to me, but it will serve as a comparison tool if we track their evolution

[larsnaesbye]

I mainly used rustup for initial setup, but good to know it’s like conda for Python.

[gwbres]

Hello @larsnaesbye

This is a little bit off topic, but do you have any recomendations in terms of GNSS receiver settings, in order to maximize Timing/Solution quality? Right now I'm investigating Elevation mask and so called "static hold" parametrization. I only have access to Ublox (8) receivers, Unfortunately we don't have access to high end Ublox (9 model) or Septentrio receivers.

[gwbres]

Hello @larsnaesbye

can you confirm that we also (possibly) want to compute 1D or even 2D differentiations, or raw pseudo ranges, and not only raw phases ?
currently I discarded anything but raw phases I think, I will improve that.

If you request a 1D or even worse a 2D diff, with the realistic context you provided last summer, the operation is very very slow on my side. I can only produce a result when focusing on 1 or 2 vehicules. But we have a good progression margin, on both record parsing and crinex decompression, so i'm expecting this to improve in a near future.

PS: Observation file generation about to be concluded :)

[larsnaesbye]

I am not quite sure about the need for 2D differentiations. I will ask my colleagues next week when they come home from autumn holidays :)

[gwbres]

I am not quite sure about the need for 2D differentiations. I will ask my colleagues next week when they come home from autumn holidays :)

Holidays are starting here, enjoy your vacation time 👍
For information, a friend of mine pointed out
this american document,
which seems to be the reference document for everyone working on RTK differential positionning for instance. I will study it, it looks like it contains all answers to my questions

[gwbres]

Observation RINEX encoding and formatting is now fully validated 👍.
I'm providing a file generation interface in the command line as of now, I'm in the process of documenting this section.

Validating V2 has honestly been a real pain.
And it looks like our parser faces a weird issue, like it's working 99.99 % of the time.
When parsing test_resources/others/rovn (temporarily moved out of pool), one epoch comprises 25 vehicules, which requires a 3 line descriptor. We know we have 25 vehicules and know we must grab 2 extra lines. But we wind up grabbing 3 lines, which destroys the 1st observations. Mistery... 👾