2022.9

[javierggt]

ska3-flight 2022.9

This PR includes the following major changes:

• backstop_history: Changes approved in FSDS 46.
• kadi: Commands-v2 will now be the default.
• ska.engarchive: Change MSIDset.filter_bad() to apply to individual MSIDs by default
• proseco:
  • Adjust acq selection to limit search hits to < 50
  • Change default OR guide and acq image size to 8x8 pixels
• aca_view: Changes/improvements to work in real-time
• astromon: new python re-implementation of older perl package for absolute astrometry.

It also includes a number of minor changes and improvements. For details, see below.

Interface Impacts:

The switch to kadi commands-v2 includes API changes that have been widely discussed in informational forums. In particular, getting commands may try using network resources by default. The commands archive files in $SKA/data will be updated to a new version upon this release.

There was a change in the default behavior when filtering bad values in cheta.fetch.MSIDset.filter_bad() (and in Msidset()): Previously, filter_bad attempted to combine the "bad" value mask of MSIDs in the set that have the same type. Now the default is to set the masks for each MSID separately.

Testing:

• HEAD.
• GRETA.

skare3 dashboard and test result password at https://icxc.cfa.harvard.edu/aspect/skare3_dash_cred.txt

The latest release candidates will be installed in /proj/sot/ska3/test on HEAD, and all release candidates will be available for testing from the usual channels:

conda create -n ska3-flight-2022.9rc7 --override-channels \
  -c https://icxc.cfa.harvard.edu/aspect/ska3-conda/flight \
  -c https://icxc.cfa.harvard.edu/aspect/ska3-conda/test \
  ska3-flight==2022.9rc7

Review

All operations critical or impacting PR's are independently and carefully reviewed. For other PR's the level of detail for review is calibrated to operations criticality. Some PR's that are confined to aspect-team-specific processing may have little to no independent review.

Deployment

ska3-flight 2022.9 will be promoted to flight conda channel and installed on HEAD and GRETA Linux upon approval of FSDS Jira ticket.

Code changes

ska3-flight changes (2022.7 -> 2022.9rc7)

New Packages

• astromon: 1.0.1
• kalman_watch: 0.1.0

Updated Packages

• aca_hi_bgd: 0.1.5 -> 0.1.6 (0.1.5 -> 0.1.6)
  • PR 12 (taldcroft): Remove sys.prefix from data_files
• aca_view: 0.9.0 -> 0.10.0 (0.9.0 -> 0.10.0)
  • PR 108 (javierggt): use a single abort queue for all child processes
  • PR 119 (javierggt): avoid np deprecated warning
  • PR 120 (javierggt): Fix multiple issues with configuration
  • PR 123 (javierggt): show the type of image telemetry
  • PR 129 (javierggt): Add settings editor, --log-level argument, and config/logging fixes
  • PR 131 (javierggt): fix fancy slider crash when there are no commands in time range
  • PR 124 (javierggt): Add text-based image widget
  • PR 130 (javierggt): Fix font and cell sizes in tables
  • PR 126 (javierggt): Major data overhaul (Ska data optional, blobs, level0/telem times)
  • PR 149 (javierggt): Small issues
  • PR 145 (javierggt): bugfix: Fix issue with missing AOKALSTR points
  • PR 152 (javierggt): serializing access to timeline inside multiprocess_server
• aca_weekly_report: 0.1.5 -> 0.1.6 (0.1.5 -> 0.1.6)
  • PR 10 (taldcroft): Update setup.py
• acdc: 4.8.0 -> 4.8.1 (4.8.0 -> 4.8.1)
  • PR 66 (taldcroft): Clean up partial processing in case of telemetry gap error
  • PR 67 (taldcroft): Only use full SRDCs with 8 quadrant readouts
  • PR 68 (taldcroft): Apply black and isort formatting; add pre-commit
• acis_taco: 4.2.1 -> 4.2.2 (4.2.1 -> 4.2.2)
  • PR 23 (taldcroft): Update setup.py
• acispy: 2.4.0 -> 2.5.0 (2.4.0 -> 2.5.0)
  • PR 11 (jzuhone): Fix ECS timing, silence Matplotlib warnings, code cleanup
• backstop_history: 1.5.3 -> 3.1.0 (1.5.3 -> 3.1.0)
  • PR 24 (jzuhone): New ECS RTS file, New method added, SCS-107 bug fixed
• chandra_aca: 4.35.0 -> 4.35.1 (4.35.0 -> 4.35.1)
  • PR 128 (taldcroft): Fixed maude_decom to work on real-time
  • PR 129 (javierggt): Bugfix: make sure maude_result argument to get_raw_aca_blobs is not modified inside the function
  • PR 130 (javierggt): fix in maude_decom.get_aca_packets from blobs
• jobwatch: 0.9.0 -> 0.10.0 (0.9.0 -> 0.9.1 -> 0.10.0)
  • PR 57 (taldcroft): Update setup.py
  • PR 58 (javierggt): Re-enable astromon checks
• kadi: 6.0.1 -> 7.0.2 (6.0.1 -> 7.0.0 -> 7.0.1 -> 7.0.2)
  • PR 244 (taldcroft): Apply Black and isort autoformatting
  • PR 245 (taldcroft): Fix a mistake in clear_caches
  • PR 246 (taldcroft): Remove dither disable from bright star hold command set
  • PR 248 (taldcroft): Major refactor/improvement in command interrupt handling
  • PR 251 (taldcroft): Fix problem where starcat_date can be uninitialized
  • PR 253 (taldcroft): Better star catalog handling
  • PR 249 (taldcroft): Make commands v2 the default
  • PR 254 (javierggt): Fix unit test failure for final v2 cmds
  • PR 256 (taldcroft): Set lookforward for Orbit event class
  • PR 255 (taldcroft): Suppress v1 FutureWarning for testing
• maude: 3.10.0 -> 3.10.4 (3.10.0 -> 3.10.1 -> 3.10.2 -> 3.10.3 -> 3.10.4)
  • PR 40 (javierggt): STATE_CODES array indices are always int
  • PR 41 (taldcroft): Handle illegal state codes in blob query
  • PR 42 (javierggt): increase tolerance in test
  • PR 43 (javierggt): Make state codes test more tolerant of time-tag changes
• mica: 4.30.1 -> 4.31.0 (4.30.1 -> 4.31.0)
  • PR 275 (taldcroft): Update setup.py
  • PR 276 (javierggt): add optional dark_cal_ids argument to dark_cal.get_dark_cal_id
  • PR 277 (taldcroft): starcheck subpackage: apply black and isort, fix line lengths, remove unused imports
• perigee_health_plots: 0.3.0 -> 0.3.1 (0.3.0 -> 0.3.1)
  • PR 22 (taldcroft): Update setup.py
• proseco: 5.7.0 -> 5.8.0 (5.7.0 -> 5.8.0)
  • PR 375 (taldcroft): Apply black and isort and update configuration files
  • PR 376 (taldcroft): Adjust acq selection to limit search hits to < 50
  • PR 377 (taldcroft): Various documentation fixes
  • PR 379 (taldcroft): Change default OR guide and acq image size to 8x8 pixels
• ska.engarchive: 4.56.0 -> 4.57.0 (4.56.0 -> 4.57.0)
  • PR 235 (taldcroft): Change MSIDset.filter_bad() to apply to individual MSIDs by default
  • PR 236 (taldcroft): Remove test_fetch_regr
  • PR 238 (taldcroft): Add warning/error to update_server_sync messages
  • PR 237 (taldcroft): Update setup.py
  • PR 240 (taldcroft): Fix server data root regex for copying server files
• ska.matplotlib: 3.14.1 -> 3.15.0 (3.14.1 -> 3.14.2 -> 3.15.0)
  • PR 25 (taldcroft): Use draw_idle instead of draw in remake_ticks
  • PR 26 (taldcroft): Add function set_min_axis_range()
• ska_helpers: 0.5.0 -> 0.6.1 (0.5.0 -> 0.6.0 -> 0.6.1)
  • PR 22 (taldcroft): Add basic_logger utility
  • PR 25 (taldcroft): bugfix: do not configure logger if it has a handler or log-level is set
• skare3_tools: 1.0.6 -> 1.0.8 (1.0.6 -> 1.0.7 -> 1.0.8)
  • PR 87 (javierggt): Improve skare3-summary and skare3-promote scripts
  • PR 89 (javierggt): fix template in skare3-changes-summary
• sparkles: 4.19.0 -> 4.20.0 (4.19.0 -> 4.20.0)
  • PR 177 (jeanconn): Silence warning from clipping MAXMAG table
• starcheck: 13.15.1 -> 13.16.0 (13.15.1 -> 13.16.0)
  • PR 388 (jeanconn): Add warning on OR if min(t_ccd) < -14
  • PR 389 (jeanconn): Skip warnings and catalog checks if no catalog - This will clean up "Uninitialized" warnings that SOT MP has been seeing.
  • PR 392 (jeanconn): Fix performance issue in get_fid_actions
  • PR 390 (jeanconn): Use kadi dither state as authoritative - This will fix situation where starcheck would exit badly on mismatched dither info after anomaly.
  • PR 391 (taldcroft): Minimal change to default to flight scenario for commands v2
  • PR 393 (jeanconn): Fix benign dither bug (Use of uninitialized value)
  • PR 394 (jeanconn): Add a Python function to get KADI_SCENARIO for printing
  • PR 395 (jeanconn): Add High IR Zone PCAD checks
• testr: 4.11.0 -> 4.11.1 (4.11.0 -> 4.11.1)
  • PR 45 (taldcroft): Update HEAD subnets to all linux machines in nodeinfo
• xija: 4.26.1 -> 4.28.1 (4.26.1 -> 4.27.0 -> 4.28.0 -> 4.28.1)
  • PR 123 (taldcroft): Improve performance of bad_times and mask_times
  • PR 130 (jzuhone): Fix problem of dropping bad_times in spec file and y-scale divergence

ska3-flight-latest changes (2022.6 -> 2022.9rc7)

New Packages

• astromon:
• fot-matlab:
• kalman_watch:

ska3-perl changes (2022.2 -> 2022.9rc7)

Updated Packages

• task_schedule: 4.2.1 -> 4.3.0 (4.2.1 -> 4.2.2 -> 4.3.0)
  • PR 17 (javierggt): Setting up automated builds
  • PR 18 (javierggt): Shiny workflow update
  • PR 19 (javierggt): Shiny workflow update
  • PR 20 (javierggt): modified workflow. This is arch-dependent
  • PR 22 (taldcroft): Allow task_schedule config in Python package

Related Issues

Fixes #900
Fixes #903
Fixes #904
Fixes #905
Fixes #906
Fixes #907
Fixes #908
Fixes #909
Fixes #910
Fixes #911
Fixes #912
Fixes #913
Fixes #914
Fixes #915
Fixes #916
Fixes #917
Fixes #918
Fixes #920
Fixes #922
Fixes #923
Fixes #924
Fixes #925
Fixes #926
Fixes #927
Fixes #928
Fixes #930
Fixes #932
Fixes #933
Fixes #934
Fixes #935
Fixes #937
Fixes #938
Fixes #939
Fixes #940
Fixes #941
Fixes #942
Fixes #943