2022.5

[javierggt]

This PR replaces #850 after I renamed branch 2022.5 to 2022.5-branch.

ska3-matlab 2022.5

This PR brings ska3-matlab in sync with ska3-flight 2022.4 (including a comprehensive update of ska3-core packages), and on top of that includes:

These are the intermediate ska3-flight releases:

• 2021.11
• 2021.12
• 2021.13
• 2021.14
• 2021.15
• 2022.1
• 2022.2
• 2022.3
• 2022.4

Interface Impacts:

ska3-flight 2021.14

kadi: add a new intermediate state to the grating states to indicate the movement time between INSR and RETR. This behavior can be disabled using kadi.command.states.disable_grating_move_duration

ska3-flight 2021.15

The acis_thermal_check interface impacts briefed in FSDS-6

ska3-flight 2022.2

Comprehensive update of ska3-core and ska3-perl meta-packages. More info here: #774

ska3-flight 2022.3

No API breaking interface impacts. Known interface changes:

• kadi:
  • For the default v1 commands archive, the output of get_cmds now includes a new time column corresponding to the date in CXC seconds.
• sparkles:
  • Non-public function sparkles.roll_optimize.allowed_rolldev has been removed.
  • Plot of star catalog for roll options now includes the star mag for potential candidate stars.
• xija_gui_fit:
  • Numerous changes/improvements to the GUI interface including new functionality (see PR 116).

ska3-flight 2022.4

• The update to kadi commands does change the star catalog command date to be the new "reference" date for star catalogs. The commands archive files (cmds2.h5, cmds2.pkl) will be updated as part of promotion and will be sync'd by users following the usual sync process.

Testing:

• 2022.5rc2 automated tests show some failures. In both cases these are tests that need fixing and do not point to any underlying issue:
  • Ska.engarchive test_fetch_derived_param_aliases. This test is known to fail some times. It is not being fixed in this release.
  • acis_thermal_check. All test_validation tests. This is due to a bug fix in xija. After the bugfix, the reference data for testing needs to be updated. It will not be updated for this release.

The latest release candidates are be installed in /proj/sot/ska3/matlab/test on HEAD and GRETA Linux. All release candidates will be available for testing from the usual channels:

conda create -n ska3-flight-2022.5rc2 --override-channels \
  -c https://icxc.cfa.harvard.edu/aspect/ska3-conda/flight \
  -c https://icxc.cfa.harvard.edu/aspect/ska3-conda/test \
  ska3-flight==2022.5rc2

Review

All operations critical or impacting PR's are independently and carefully reviewed. For other PR's the level of detail for review is calibrated to operations criticality. Some PR's that are confined to aspect-team-specific processing may have little to no independent review.

Deployment

ska3-matlab 2022.5 will be promoted to flight conda channel and installed on GRETA Linux after approval from FOT team.

Code changes

These are only the ska3-matlab changes. For the ska3-core changes see the interface impacts above.

ska3-matlab changes (2021.10 -> 2022.5rc3)

New Packages

• fot-matlab

Removed Packages

• acisfp_check
• bep_pcb_check
• dea_check
• dpa_check
• fep1_actel_check
• fep1_mong_check
• psmc_check

Updated Packages

• aca_view: 0.7.2 -> 0.9.0 (0.7.2 -> 0.8.0 -> 0.9.0)
  • PR 84 (javierggt): Commands in time slider and other small changes
  • PR 85 (javierggt): Refactored config to play nice with QSettings.
  • PR 86 (javierggt): Add MAUDE channel as an option
  • PR 91 (javierggt): Use first kadi maneuver to set obsid start, not last
  • PR 93 (javierggt): fixed wrong logging message
  • PR 92 (javierggt): Fix the way the image view slides as dither pushes the image against the edge
  • PR 90 (javierggt): Fix issue ACIS updates approved 31-Aug-2018 #88.
  • PR 96 (javierggt): Cleanup (inconsequential changes)
  • PR 97 (javierggt): Set font sizes according to window size
  • PR 99 (javierggt): Shutdown cleanup
  • PR 101 (javierggt): Toolbars
  • PR 100 (javierggt): Small fixes
• acis_taco: 4.2.0 -> 4.2.1 (4.2.0 -> 4.2.1)
  • PR 21 (javierggt): remove use of deprecated np.int
• acis_thermal_check: 3.6.0 -> 4.2.0 (3.6.0 -> 4.0.0 -> 4.1.0 -> 4.1.1 -> 4.1.2 -> 4.2.0)
  Includes the following merges:
  • PR 41: Update for SCS-106 change and new power commands
  • PR 42: Write .hist files to --outdir directory
  • PR 43: Temporarily disable the inclusion of gratings moves states
  • PR 44 (jzuhone): Move all of the temperature model scripts into the main package
  • PR 45 (jzuhone): Use limits from the JSON file, refactor prediction plots
  • PR 46 (jzuhone): Prevent matplotlib warning
  • PR 48 (jzuhone): Don't check the OCAT for "hot" ACIS observations if there are no observations (vehicle loads)
• acispy: 2.2.0 -> 2.4.0 (2.2.0 -> 2.3.0 -> 2.4.0)
  • PR 9 (jzuhone): Fix issues with thermal model handling
  • PR 10 (jzuhone): Use thermal limits from JSON files, support ACIS FP in SimulateECSRun, new "attitude" argument
• agasc: 4.11.4 -> 4.12.0 (4.11.4 -> 4.11.5 -> 4.11.6 -> 4.11.7 -> 4.12.0)
  • PR 129 (taldcroft): Add comments to obs_status.yml with process help
  • PR 136 (javierggt): fix breakage due to kadi 5.9.1 changing dtypes
  • PR 133 (javierggt): Fix matplotlib ticks warning
  • PR 132 (javierggt): Do not count non-npm times when calculating f_ok
• annie: 0.11.0 -> 0.11.1 (0.11.0 -> 0.11.1)
  • PR 109 (javierggt): remove use of deprecated np.float and np.int
• backstop_history: 1.4.1 -> 1.5.3 (1.4.1 -> 1.5.0 -> 1.5.1 -> V1.5.2 -> 1.5.3)
  • PR 20 (Gregg140): Update for SCS-106 change and new power commands
  • PR 21 (Gregg140): Use cr backstop files
  • PR 22 (Gregg140): Now processes a full continuity (science + vehicle commands) load as …
  • PR 23 (Gregg140): Remove unnecessary and incorrectly applied round off
• chandra.cmd_states: 3.16.0 -> 3.17.0 (3.16.0 -> 3.17.0)
  • PR 64 (taldcroft): Add dither cmd sets and fix nsm cmd_set, etc
• chandra.time: 4.0.0 -> 4.0.1 (4.0.0 -> 4.0.1)
  • PR 53 (taldcroft): Allow bytes input to date2secs
• chandra_aca: 4.33.0 -> 4.35.0 (4.33.0 -> 4.34.0 -> 4.34.1 -> 4.34.2 -> 4.34.3 -> 4.35.0)
  • PR 119 (jeanconn): Update guide_count for 5.2mag bright limit
  • PR 120 (javierggt): remove use of deprecated np.bool and np.int
  • PR 121 (jeanconn): Use has_internet to skip planet tests requiring network
  • PR 122 (javierggt): Fix tests after changes in maude
  • PR 124 (taldcroft): Allow for broadcastable array inputs in snr_mag_for_t_ccd and guide_count
• cxotime: 3.2.5 -> 3.3.0 (3.2.5 -> 3.2.6 -> 3.3.0)
  • PR 26 (taldcroft): Remove debug print statements
  • PR 27 (taldcroft): Add date2secs for fast conversion of Year DOY date to CXC seconds
  • PR 28 (taldcroft): Add secs2date function and speed up date2secs
• find_attitude: 3.4.0 -> 3.4.2 (3.4.0 -> 3.4.1 -> 3.4.2)
  • PR 16 (jeanconn): Fix tests
  • PR 18 (taldcroft): Silence the sherpa warning about DS9 not being available
• fot-matlab: 1.0.0 -> 2.0.1 (1.0.0 -> 2.0.1)
  • PR 10 (jskrist): Matlab 11834
  • PR 11 (jskrist): Create kadi_utils.py
  • PR 12 (jskrist): fixed PEP8 syntax violations
  • PR 13 (jskrist): Updated np.array() call to include explicit dtype argument
  • PR 14 (jskrist): Acis fp sun eph support
  • PR 15 (jskrist): forced all state times to be dtype=float64.
• jobwatch: 0.2.0 -> 0.7.0 (0.2.0 -> 0.3.0 -> 0.4.0 -> 0.5.0 -> 0.6.0 -> 0.7.0)
  • PR 48 (taldcroft): Check that the last kadi dwell < 3 days old
  • PR 49 (jeanconn): Exclude some mica errors from log warnings
  • PR 50 (taldcroft): Split hourly watch into Ska and MTA jobs
  • PR 51 (jeanconn): Add more workflows to this project
  • PR 53 (jeanconn): Remove some defunct checks
  • PR 54 (jeanconn): Remove check on sybase load_segments
• kadi: 5.6.0 -> 5.11.0 (5.6.0 -> 5.8.0 -> 5.8.1 -> 5.9.0 -> 5.9.1 -> 5.9.2 -> 5.10.0 -> 5.11.0)
  • PR 210 (taldcroft): Make changes for grating move duration (again)
  • PR 214 (javierggt): remove use of deprecated np.str
  • PR 215 (javierggt): fix warning
  • PR 212 (taldcroft): Implement commands archive v2.0 (beta release)
  • PR 218 (javierggt): Fix regressions from v2 (Update kadi to version > 3.16.4 #212)
  • PR 219 (taldcroft): Update homepage URL
  • PR 224 (taldcroft): Allow outputting all keys in reduce_states
  • PR 229 (taldcroft): Remake last two years of MajorEvent table each time
  • PR 221 (taldcroft): Add starcat_date to observation parameters + fix caching and minor issues
  • PR 232 (taldcroft): Allow OCCweb access using noodle path
  • PR 233 (taldcroft): Fix lucky testing and occweb lucky functionality
• maude: 3.7.0 -> 3.9.0 (3.7.0 -> 3.8.0 -> 3.9.0)
  • PR 36 (taldcroft): Update MSID state code and index files
  • PR 37 (taldcroft): Fix get MSID in chunks and use CxoTime
  • PR 38 (taldcroft): Support parsing of binary blobs (fmt=0) + new blobs_to_arrays() function
  • PR 39 (taldcroft): Add support for allpoints, hr, nearest, icalc, ircts params and improve chunked queries
• mica: 4.27.1 -> 4.29.0 (4.27.1 -> 4.27.2 -> 4.28.0 -> 4.29.0)
  • PR 266 (jeanconn): Fix not track percent so it is not track percent
  • PR 265 (jeanconn): Retry aca0 web query and ignore gateway errors in log checks
  • PR 264 (taldcroft): Make a target table product
  • PR 267 (taldcroft): Add CDA web services API and expand functionality of Ocat target target query
  • PR 270 (jeanconn): Update ocat target table first in cron task
  • PR 271 (jeanconn): Use CxoTime and fix time range for update
  • PR 269 (javierggt): Remove use of Ska.Shell
  • PR 272 (taldcroft): Improve speed of getting local OCAT data
• parse_cm: 3.7.1 -> 3.9.0 (3.7.1 -> 3.8.0 -> 3.9.0)
  • PR 36 (taldcroft): Fix typo in docs and emphasize deprecation
  • PR 35 (taldcroft): Generate backstop commands from DOT or FOT request
  • PR 37 (taldcroft): Give write_backstop function the ability to output to a file-like object
• perigee_health_plots: 0.2.1 -> 0.3.0 (0.2.1 -> 0.3.0)
  • PR 21 (taldcroft): Use chandra_models for planning limit
• proseco: 5.3.0 -> 5.6.0 (5.3.0 -> 5.4.0 -> 5.5.0 -> 5.6.0)
  • PR 368 (jeanconn): Move bright limits to approach or hit the new 5.2 mag onboard limit
  • PR 370 (jeanconn): Exclude guide star if it overlaps with brighter/better guide star
  • PR 371 (javierggt): Prevent matplotlib warning
  • PR 372 (taldcroft): Allow bonus faint guide stars for dyn bgd enabled
• quaternion: 4.0.1 -> 4.1.0 (4.0.1 -> 4.1.0)
  • PR 39 (taldcroft): Add from_attitude class method for more permissive init
• ska.arc5gl: 3.1.5 -> 3.2.0 (3.1.5 -> 3.2.0)
  • PR 4 (taldcroft): Allow arc5gl user to be configured via ~/.arcgl_user file
• ska.engarchive: 4.53.1 -> 4.55.2 (4.53.1 -> 4.54.0 -> 4.54.1 -> 4.55.0 -> 4.55.1 -> 4.55.2)
  • PR 223 (taldcroft): Improve MSID representation
  • PR 224 (taldcroft): Try preventing square overflow and give debug info otherwise
  • PR 225 (javierggt): remove use of deprecated np.bool
  • PR 226 (taldcroft): Fix pickling for MSIDset and add tests
  • PR 227 (taldcroft): Allow CALC_ prefix as an alias for DP_
  • PR 228 (taldcroft): Fix integration tests
  • PR 230 (javierggt): Fix test failure from updated MUPS model
• ska.matplotlib: 3.12.0 -> 3.14.1 (3.12.0 -> 3.13.0 -> 3.13.1 -> 3.14.0 -> 3.14.1)
  • PR 19 (taldcroft): Handle shaped and zero-length input data in cxctime2plotdate
  • PR 20 (javierggt): remove use of deprecated np.float
  • PR 21 (javierggt): Prevent matplotlib warning
  • PR 24 (taldcroft): Fix plot_cxctime for compatibility with matplotlib 3.5
• ska.numpy: 3.9.0 -> 3.9.1 (3.9.0 -> 3.9.1)
  • PR 14 (javierggt): remove use of deprecated np.int
• ska.sun: 3.7.0 -> 3.8.1 (3.7.0 -> 3.8.0 -> 3.8.1)
  • PR 19 (jeanconn): Move allowed_rolldev here from sparkles and update pitch/roll table
  • PR 21 (taldcroft): Fix mistake in docs and update module tagline
• ska3-core: 2021.8 -> 2022.3
• ska_sync: 4.9.1 -> 4.10.0 (4.9.1 -> 4.10.0)
  • PR 30 (taldcroft): Update ska_sync_config for ocat_target_table.h5
  • PR 29 (taldcroft): Changes to the default ska_sync_config
  • PR 31 (jeanconn): Stop syncing old ska2 kadi events and old cmd_states
• skare3_tools: 1.0.4 -> 1.0.6 (1.0.4 -> 1.0.5 -> 1.0.6)
  • PR 83 (javierggt): Fix version overwrite when building
  • PR 85 (javierggt): Fix: Enable releases with closed PRs
  • PR 84 (javierggt): change conda channels in default config to what they should be
• sparkles: 4.12.2 -> 4.17.0 (4.12.2 -> 4.13.0 -> 4.13.1 -> 4.14.0 -> 4.15.0 -> 4.16.0 -> 4.17.0)
  • PR 162 (jeanconn): Update bright limit check to use 5.2 mag
  • PR 163 (jeanconn): Update roll review test for up to 5.2 mag stars
  • PR 164 (taldcroft): Allow specifying all roll options from command line
  • PR 161 (taldcroft): Add module to find ER catalog
  • PR 165 (taldcroft): Fix ER attitude finder test fails due to 5.2 mag limit change
  • PR 166 (jeanconn): Add a check for two tracked things that are within 12 pixels.
  • PR 168 (jeanconn): Use allowed_rolldev in Ska.Sun
  • PR 169 (taldcroft): Add mag for candidates + fix minor plot() bug
  • PR 171 (taldcroft): Allow opening a pickle from Noodle/OCCweb
  • PR 172 (taldcroft): Apply dyn bgd bonus in computing guide_count
• starcheck: 13.12.0 -> 13.15.1 (13.12.0 -> 13.13.0 -> 13.14.0 -> 13.15.0 -> 13.15.1)
  • PR 377 (jeanconn): Move the bright limit to 5.2 in code and checklist
  • PR 378 (jeanconn): round guide count and P2
  • PR 379 (jeanconn): Add missing newline to guide count warn
  • PR 380 (jeanconn): Add track overlap check
  • PR 381 (javierggt): prevent matplotlib warning
  • PR 382 (jeanconn): Set linestyle via fmt in plot_two
• testr: 4.6.0 -> 4.10.0 (4.6.0 -> 4.7.0 -> 4.8.0 -> 4.9.0 -> 4.10.0)
  • PR 38 (taldcroft): Add has_internet function to test_helpers
  • PR 37 (javierggt): Ignore deprecation warning that comes from external packages
  • PR 39 (taldcroft): Check internet connectivity by requesting web URL
  • PR 41 (taldcroft): Disable hypothesis plugin autoload by default
  • PR 42 (javierggt): Extirpate Ska.Shell
  • PR 43 (taldcroft): Add Coverage
• xija: 4.24.0 -> 4.26.1 (4.24.0 -> 4.24.1 -> 4.25.0 -> 4.26.0 -> 4.26.1)
  • PR 115 (javierggt): remove use of deprecated np.float and np.int
  • PR 116 (taldcroft): Updates for xija_gui_fit
  • PR 119 (taldcroft): Fix epoch interpolation bug, wrong function name for interpolation
  • PR 117 (taldcroft): Adjust earthshine effect based on Earth phase
  • PR 121 (taldcroft): Updated 'Hack' to only overwrite final prediction mvals instead of all mvals + PEP8

Related Issues

Fixes #838
Fixes #840
Fixes #842
Fixes #843
Fixes #844
Fixes #845
Fixes #846
Fixes #848
Fixes #849