Fix computed Comp_Quat class and close pickle files

[taldcroft]

Description

The Computed_Quat computed MSID is broken when there are actual bad data in the fetch range. The resultant Msid object has vals that are not filtered. This happens for samples at ['2024:064:09:27:02.652' '2024:064:09:27:03.677']. This problem has stopped updates to the Kalman watch perigee monitor window data page since around day 063.

The root cause of the problem was checking that a computed attribute (including times, vals, bads, units) is a subclass of ndarray. For the computed quaternion this is not the case, instead it is just follows (somewhat) the array protocol. For the purposes here I just check for a shape attribute.

This includes a change that was required by the new test. The goal was to show that no warnings are issued when fetching data over the impacted time range. However, the idiom of reading a pickle file with pickle.load(open(filename, "rb")) issues a ResourceWarning. This is a bit funky because pytest normally ignores that (probably because it is so common), but anyway it was getting in the way of the test I wrote, which technically passed but gave the warning below. This is just a mess so I decided to fix the source of the problem.

cheta/tests/test_comps.py .                                                                                         [100%]

==================================================== warnings summary =====================================================
cheta/cheta/tests/test_comps.py::test_quat_comp_bad_times
  /Users/aldcroft/miniconda3/envs/ska3/lib/python3.11/site-packages/_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>
  
  Traceback (most recent call last):
    File "/Users/aldcroft/git/cheta/cheta/fetch.py", line 300, in load_msid_names
      all_colnames[k] = pickle.load(open(os.path.join(*msid_names_file), "rb"))
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  ResourceWarning: unclosed file <_io.BufferedReader name='/Users/aldcroft/ska/data/eng_archive/data/dp_eps16/colnames.pickle'>
  
    warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

Interface impacts

None

Testing

Unit tests

• Mac

(ska3) ➜  cheta git:(fix-quat-comp) git rev-parse HEAD                   
33660ad6828420e3bbf3dbef97522e6d4aeefe20
(ska3) ➜  cheta git:(fix-quat-comp) pytest
=================================================== test session starts ===================================================
platform darwin -- Python 3.11.8, pytest-7.4.4, pluggy-1.4.0
rootdir: /Users/aldcroft/git
configfile: pytest.ini
plugins: timeout-2.2.0, anyio-4.3.0
collected 174 items                                                                                                       

cheta/tests/test_comps.py ............................................................                              [ 34%]
cheta/tests/test_data_source.py .........                                                                           [ 39%]
cheta/tests/test_fetch.py ................................                                                          [ 58%]
cheta/tests/test_intervals.py .........................                                                             [ 72%]
cheta/tests/test_orbit.py .                                                                                         [ 72%]
cheta/tests/test_remote_access.py ......                                                                            [ 76%]
cheta/tests/test_sync.py ........                                                                                   [ 81%]
cheta/tests/test_units.py ...........                                                                               [ 87%]
cheta/tests/test_units_reversed.py ...........                                                                      [ 93%]
cheta/tests/test_utils.py ...........                                                                               [100%]

Independent check of unit tests by Jean

• Linux

(ska3-flight-2024.1rc4) jeanconn-fido> pytest
================================================================== test session starts ==================================================================
platform linux -- Python 3.11.8, pytest-7.4.4, pluggy-1.4.0
rootdir: /proj/sot/ska/jeanproj/git
configfile: pytest.ini
plugins: timeout-2.2.0, anyio-4.3.0
collected 174 items                                                                                                                                     

cheta/tests/test_comps.py ............................................................                                                            [ 34%]
cheta/tests/test_data_source.py .........                                                                                                         [ 39%]
cheta/tests/test_fetch.py ................................                                                                                        [ 58%]
cheta/tests/test_intervals.py .........................                                                                                           [ 72%]
cheta/tests/test_orbit.py .                                                                                                                       [ 72%]
cheta/tests/test_remote_access.py ......                                                                                                          [ 76%]
cheta/tests/test_sync.py ........                                                                                                                 [ 81%]
cheta/tests/test_units.py ...........                                                                                                             [ 87%]
cheta/tests/test_units_reversed.py ...........                                                                                                    [ 93%]
cheta/tests/test_utils.py ...........                                                                                                             [100%]

============================================================ 174 passed in 318.88s (0:05:18) ============================================================
(ska3-flight-2024.1rc4) jeanconn-fido> git rev-parse HEAD
33660ad6828420e3bbf3dbef97522e6d4aeefe20

Functional tests

In HEAD flight ska3 I ran this to https://icxc.cfa.harvard.edu/aspect/test_review_outputs/cheta/cheta-pr260/kalman_flight/
which shows Kalman data for the last year and that the monitor window job did not create output (error).

kalman_watch_low_kalman_mon --data-dir /proj/sot/ska/www/ASPECT_ICXC/test_review_outputs/cheta/cheta-pr260/kalman_flight --lookback 360
kalman_watch_kalman_perigee_mon --make-html  --data-dir /proj/sot/ska/www/ASPECT_ICXC/test_review_outputs/cheta/cheta-pr260/kalman_flight --lookback 360
kalman_watch_monitor_win_perigee  --data-dir /proj/sot/ska/www/ASPECT_ICXC/test_review_outputs/cheta/cheta-pr260/kalman_flight 

PR output to https://icxc.cfa.harvard.edu/aspect/test_review_outputs/cheta/cheta-pr260/kalman_pr/ with same Kalman output except monitor plot made.

kalman_watch_low_kalman_mon --data-dir /proj/sot/ska/www/ASPECT_ICXC/test_review_outputs/cheta/cheta-pr260/kalman_pr --lookback 360
kalman_watch_kalman_perigee_mon --make-html  --data-dir /proj/sot/ska/www/ASPECT_ICXC/test_review_outputs/cheta/cheta-pr260/kalman_pr --lookback 360
kalman_watch_monitor_win_perigee  --data-dir /proj/sot/ska/www/ASPECT_ICXC/test_review_outputs/cheta/cheta-pr260/kalman_pr

[jeanconn]

LGTM and I extended the functional testing with the on-the-side Kalman Watch. I also note that I recall seeing the pickle-not-closed issue in non-pytest-test work with cheta recently so that fix makes sense to me as well.