New star catalog with Gaia DR3

[henrysky]

Description

Still draft and writing. See discussion in #3982

Major Changes:

• Star catalogs are built upon Hipparcos and Gaia DR3. Every stars brighter than magnitude ~6.5 should be included for visual purposes. In total having ~220 millions stars down to ~18 mag.
• New star catalogs format and codes to allow proper astrometry calculation. RA, DEC, PMRA, PMDEC, Parallax, Radial Velocity (and apperant magnitude) are calculated and displayed at epoch T for stars that have all required data.
• RA/DEC coordinates are now stored at ICRS epoch J2016 (so one does not need to propagate large amount of Gaia stars back to J2000 but Hippracos stars are forwarded to J2016), there is now epoch information at the header of each catalog so Stellarium can check if the epoch is the one expected.
• Magnitude and color system remains the same as the old catalogs, meaning now V-band magnitudes for majority of the stars are estimated from Gaia G-band magnitude and G_bp-G_rp color with about ~0.03 mag scattering (see https://gea.esac.esa.int/archive/documentation/GDR3/Data_processing/chap_cu5pho/cu5pho_sec_photSystem/cu5pho_ssec_photRelations.html). B-V color are more challenging if one simply estimating from G-mag and G_bp-G_rp color as shown. As a result, the values in the new catalog are synethetic photometry computed from Gaia low resolution spectra (~0.03 mag scattering compared among Hipparcos stars).
• Star1 in the catalog now has object types field and will be displayed among the field Type on top of existing information. Meaning of the display in its current state can be found at https://simbad.cds.unistra.fr/Pages/guide/otypes.htx.
• Star1 that travel outside of their own zone in the far future/past are handled by pre-computing which stars will have this behavior and put them in a global zone (an extra zone for each geodesic level). So stars like Arturus can be displayed properly in the far future.
• Some stars being difficult to be selected are fixed.
• Search by Gaia source ID are added (both via SIMBAD and offline), so the star with that Gaia source ID will be selected instead of a custom object
• Updated documentation regarding the star catalogs and astrometry accuracy.

I will upload the code to generate the new catalog here: https://github.com/henrysky/stellarium_star_catalogs. Give me a few days to organize.

To-do list for Henry:

• Release source code

Fixes #18
Fixes #85
Fixes #221
Fixes #348
Fixes #362
Fixes #363
Fixes #364
Fixes #369
Fixes #398
Fixes #399
Fixes #677
Fixes #777
Fixes #1016
Fixes #1032
Fixes #1061
Fixes #1189
Fixes #1192
Fixes #1541
Fixes #1659
Fixes #1724
Fixes #2786
Fixes #3108
Fixes #3136
Fixes #3315

Many issues related to missing stars, false stars, incorrect information, astrometry inconsistent with other softwares...

Screenshots (if appropriate):

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• This change requires a documentation update
• Housekeeping

How Has This Been Tested?

Test Configuration:

• Operating system: MacOS 15
• Graphics Card: Apple M3

Checklist:

• My code follows the code style of this project.
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation (header file)
• I have updated the respective chapter in the Stellarium User Guide
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged and published in downstream modules

[gzotti]

Just running on RPi4. Mem footprint is very relaxed (comparable to the old). I don't think I had ever downloaded all star catalogs. Now I start seeing some loss in framerate at level 7 of 9. 12fps is still acceptable, though. I dare going still deeper...
What is the recommended way to remove a star catalog? Or could we just deactivate somehow? I guess only manually, editing starsConfig.json?

And important for comparisons against older versions or if there is just other needs to run older versions: Is it possible to keep all catalogs under one Stellarium user dir? The star catalog's name is always "default" and has been probably since 2006. If there is need for separate catalogs, this new one could be named e.g. hip_gaia, so that only somewhere in code (StarMgr.cpp) the directory name is adjusted from default to hip_gaia. Note that I don't say we must allow loading the old catalog into new program versions by config, but this is such a significant change that we should think about such mixed legacy installations.

[alex-w]

Just running on RPi4. Mem footprint is very relaxed (comparable to the old). I don't think I had ever downloaded all star catalogs. Now I start seeing some loss in framerate at level 7 of 9. 12fps is still acceptable, though. I dare going still deeper... What is the recommended way to remove a star catalog? Or could we just deactivate somehow? I guess only manually, editing starsConfig.json?

Just remove specific *.cat files - no need manually editing json file.

And important for comparisons against older versions or if there is just other needs to run older versions: Is it possible to keep all catalogs under one Stellarium user dir? The star catalog's name is always "default" and has been probably since 2006. If there is need for separate catalogs, this new one could be named e.g. hip_gaia, so that only somewhere in code (StarMgr.cpp) the directory name is adjusted from default to hip_gaia. Note that I don't say we must allow loading the old catalog into new program versions by config, but this is such a significant change that we should think about such mixed legacy installations.

JSON file will be recreated if their version number is not equal to expected one in the binary package - as result you may use one place to storage all possible catalogs ;)

[alex-w]

Actually for @alex-w: I haven't done so in a long time now. After loading a new star catalog it appears the stars are immediately used. At the end of catalog downloads however, the message is something like "All catalogs downloaded, restart Stellarium to load them". I think this last sentence could be removed, right?

Maybe, I just didn’t tested loading and rendering catalogs on-the-fly…

[gzotti]

OK, but running e.g. 25.1, 1.2 and 0.15.2 on the same PC (for comparison, linked with RemoteSync or for whatever reasons; or alternating between versions for with/without ASCOM stuff), sharing the same STEL_USERDIR, may trigger conflicts when the second launched overwrites the first's JSON. (And does this mechanism work for old versions like 0.15?) This looks like dangerous design for me, and goes against whatever may have been planned in 2010 when "default" was apparently introduced. @xalioth, @10110111 ?

[alex-w]

OK, but running e.g. 25.1, 1.2 and 0.15.2 on the same PC (for comparison, linked with RemoteSync or for whatever reasons; or alternating between versions for with/without ASCOM stuff), sharing the same STEL_USERDIR, may trigger conflicts when the second launched overwrites the first's JSON. (And does this mechanism work for old versions like 0.15?)

Overwriting is possible after comparing versions in JSON file and in binary package, plus our catalogs is not changed frequently…

This looks like dangerous design for me, and goes against whatever may have been planned in 2010 when "default" was apparently introduced. @xalioth, @10110111 ?

Well, the default section for our catalogs (stars + dso) and some custom catalogs was ambitious idea, but it never implemented and the code for switching between various collections was removed a long time ago…

[gzotti]

OK, no need to add switching code. Does a simple change of the catalog path in StarMgr::copyDefaultConfigFile() and StarMgr::init() lead to the suggested separation of "all versions which used the HIP/TYC/NOMAD catalog (named "default" from 2010 to 2024)" and "versions starting with 24.4+ using this new HIP/GAIA catalog". Old versions keep working with their current logic (in theory even catalog bugs may still be fixed, as unlikely as this is), while new versions will just never have the necessity to touch the "default" catalog. If you have no old version installed, the name of the (only) catalog will just be hip_gaia (or whatever we name this now, "v25", just not "default"). Truly incompatible changes should be handled like this, IMHO.

[alex-w]

OK, no need to add switching code. Does a simple change of the catalog path in StarMgr::copyDefaultConfigFile() and StarMgr::init() lead to the suggested separation of "all versions which used the HIP/TYC/NOMAD catalog (named "default" from 2010 to 2024)" and "versions starting with 24.4+ using this new HIP/GAIA catalog". Old versions keep working with their current logic (in theory even catalog bugs may still be fixed, as unlikely as this is), while new versions will just never have the necessity to touch the "default" catalog. If you have no old version installed, the name of the (only) catalog will just be hip_gaia (or whatever we name this now, "v25", just not "default"). Truly incompatible changes should be handled like this, IMHO.

Maybe you're right and string stars/default in the code (C++ & CMake) should be changed to stars/hip_gaia

[alex-w]

Actually for @alex-w: I haven't done so in a long time now. After loading a new star catalog it appears the stars are immediately used. At the end of catalog downloads however, the message is something like "All catalogs downloaded, restart Stellarium to load them". I think this last sentence could be removed, right?

Just checked right now - no, we shouldn't remove this last sentence, because stars are not loaded from catalogs on-the-fly (at the moment of loading catalog)

[gzotti]

RPi3 also works, at least with extra cats stars_4 and stars_5. It downloads stars_6 but initialisation fails due to lack of memory. I did not realize this and happily started downloading even stars_7, then stellarium crashed with std::bad_alloc. But I think RPi3 is just the minimum platform to be used for limited installations, so going to about Pluto magnitude (13.75) should be fine.

[henrysky]

I am going to fix issue of the two stars brought up by @Atque. @alex-w Should I bump all the star catalogs version to 1.0? Beacuse the new star catalogs are not compatible with the old v0 star catalogs? (and thanks for fixing the guide!)

[alex-w]

@alex-w Should I bump all the star catalogs version to 1.0? Beacuse the new star catalogs are not compatible with the old v0 star catalogs?

I see no necessary for just renaming catalogs

[henrysky]

I have fixed all missing pm/parallax issue mentioned by others. @alex-w Only stars4/5 is changed and is here https://www.astro.utoronto.ca/~hleung/shared/stellarium_new_catalog/trial5/.

[alex-w]

I have fixed all missing pm/parallax issue mentioned by others. @alex-w Only stars4/5 is changed and is here https://www.astro.utoronto.ca/~hleung/shared/stellarium_new_catalog/trial5/.

OK, done!

[alex-w]

@gzotti @10110111 any comments?

[10110111]

Some new files here lack the POSIXly-desired trailing newlines, which is indicated by git-diff and by GitHub's diff viewer. These should be added.

P.S. A good question - squash or rebase?

Are there any testable versions among these commits, so that it would make sense to run, say, git-bisect on them? If yes, I suppose it's better to rebase.

In any case, the changes to documentation (the Guide first of all) should be separated and squashed into a single commit after all the others.

[gzotti]

Sorry our ISP was offline for some time in mid-afternoon for whatever reason. Dec.24 overload? (about 15.000 reports at my ISP.)

The current incremented version number is OK for me. Just the new path was important to allow parallel installations, thanks.
I just wanted to follow the morphing of our 3-star-linked constellation artwork over the millennia. However it seems the projection is computed only at loading of the respective skyculture. E.g. loading Inuit around 85000AD looks weird. But this is not for this PR. Now SC authors face a new challenge: define the 3-star link with low-PM stars :-) (Or make distortion part of the show...)

I think there is no half-working in-between version to ever bisect. 24.4 has the HIP/TYC/NOMAD catalog, and this PR switches to HIP/GAIA DR3. For me a rebase&squash would be OK, even if the changes is large. Also, the directory issue may break any bisects. Of course, later fixes will be separate. So, for me it's OK.

[gzotti]

Cosmetics like the empty line at end should be fixed, I second that (whatever git or CodeFactor report).

Else, this looks OK for me, many thanks!