-
Notifications
You must be signed in to change notification settings - Fork 4
Ska3 prime (2023.1 with Python 3.10) testing
This document describes testing plans and results for the transition to Ska3-prime. This is version 2023.1 with Python 3.10 and updates to most of the core packages such as NumPy and Astropy.
Because of the significant and sweeping changes in the core packages that comprise Ska3, teams that use Ska3 as part of flight operations are responsible for the following:
- Performing an audit to identify critical tools or software that depend on Ska3
- Testing those tools using the Ska3 prime environment
- Reporting results by directly editing this page, either with the results in full or by inserting a link to a separate test results page.
Forward and Fix
It is worth noting that many tools are not flight critical and being broken for a week or two is tolerable. In those cases the most efficient path may be "forward and fix". In other words, plan to wait until the promotion of Ska3 shiny to flight and then test at that time. This can be a useful strategy for tools where it is difficult to perform independent testing, e.g. if there are hardwired data paths.
Even in these cases it is important to identify in advance such tools along with a post-promotion test plan. "Testing" will likely consist of running the tool and evaluating the outputs for correctness.
Core testing is done via the Ska3 integration testing package which consists of running available package unit tests and regression test scripts using machinery in the testr package. This is done with the run_testr command, where some tests may be skipped on standalone platforms.
Current results for 2023.1rc12
See the test outputs in the PR at https://github.com/sot/skare3/pull/1008
- Load review tools pass all tests (unit/regression)
- sparkles/proseco pass all unit tests. - JC on fido 3/20/2023
- starcheck - ran full regression test set with /proj/sot/ska3/test with starcheck version 14.0.2 (in test and release). No diffs. -JC 03/17/2023
Comments - Reviewed all MP code/scripts and found that we are not using any Ska packages that were renamed. All other updates look good.
- D. Patnaude Jira review - ran test version on HEAD network - no issues found with HRC CEA thermal checking tool.
MTA is still using a private version of SKA. We will commence our transition to flight SKA with this version,
SOT MP relies on starcheck and acisfp_check as part of our load review process. In addition, sparkles.run_one_yoshi() is used to assess ACA star fields in advance of long-term scheduling as well as helping to inform proposers if a given target will present difficulties.
We tested the 2023.1rc12 release located at /proj/sot/ska3/test in our load review process. Both starcheck and acisfp_check ran successfully and created the expected output. We note that the starcheck results were different when running against the MAR1323 schedule, but this is to be expected.
Additionally, we installed 2023.1rc12 at /data/mpcrit5/ska3-dev and used this installation to run our code to evaluate ACA star fields. The code ran as expected.
This release is okay for SOT MP. To be updated for ska3-prime by SOT MP.
Run with
c3po-v$ cd ~/git/acdc
c3po-v$ mkdir -p regress_prime/VC2
# Get enough data to make the last two reports
c3po-v$ cp /dsops/GOT/input/2023*VC2* regress_prime/VC2
c3po-v$ cp /dsops/GOT/input/2022_3*VC2* regress_prime/VC2
c3po-v$ python -m acdc.process_vc2_to_l0 --vc2-root=regress_prime/VC2 --data-root=regress_prime
c3po-v$ python -m acdc.process_l0_to_quads --data-root=regress_shiny --start 2022:339
# I then edited regress_prime/vc2_index.dat to exclude early srdcs and ones before safe mode
# recovery as excluded. The intent just to get 10 that line up with what was processed in flight
c3po-v$ python -m acdc.process_quads_to_cals --data-root=regress_prime
c3po-v$ python -m acdc.make_reports --data-root=regress_prime
Check results:
- All processing runs successfully with reasonable outputs
- Output report for 2023:023 cal looks reasonable
- Output calibration image is consistent with flight (with note)
Note on comparing outputs:
The temperature values assigned to cals varied in this process by up to 0.08 C compared to flight. This is likely because flight processing is done soon after the cal and can be using available MAUDE telemetry from realtime. This will generate differences in temperatures and mask values because the temperatures are not available during the VC2 readout interval.
For the 2023:023 cal the diffs are:
- Mean: -0.010 e-/sec
- Stddev: 0.20 e-/sec
- Max: 27 e-/sec for a pixel with dark cal 176 e-/sec
In [5]: dat1 = fits.open("023/cal_2023_023_08_42_45.fits.gz")[0].data
In [6]: dat2 = fits.open("023/flight_report/023/cal_2023_023_08_42_45.fits.gz")[0].data
In [7]: diff = dat1 - dat2
In [11]: np.max(np.abs(diff))
Out[11]: 27.136017
In [12]: np.mean(diff)
Out[12]: -0.00981138
In [13]: np.std(diff)
Out[13]: 0.2013503
In [14]: np.argmax(np.abs(diff))
Out[14]: 425343
In [15]: dat1.ravel()[425343]
Out[15]: 176.86723
In [16]: dat2.ravel()[425343]
Out[16]: 149.73122
Reprocess MAR2023 with Ska3-flight and Ska3-prime:
cd ~/git/acis_taco
mkdir tmp/acis_taco
cd cd tmp/acis_taco
mkdir flight
mkdir prime
/proj/sot/ska3/flight/bin/skare /proj/sot/ska3/flight/share/acis_taco/make_esaview_data.py --nweeks=1 --data-root=flight
/proj/sot/ska3/test/bin/skare /proj/sot/ska3/test/share/acis_taco/make_esaview_data.py --nweeks=1 --data-root=prime
Then from ipython
import pickle
dat1 = pickle.load(open('tmp/acis_taco/flight/MAR2023.pkl', 'rb'))
dat2 = pickle.load(open('tmp/acis_taco/prime/MAR2023.pkl', 'rb'))
np.all(dat1['illums'] == dat2['illums']) # True
np.all(dat1['times'] == dat2['times']) # True
Prime test plan:
- Run aca_hi_bgd_update "live" into the real data/web area with shiny test environment
- Confirm no errors from the script
- Confirm output main page at https://cxc.cfa.harvard.edu/mta/ASPECT/aca_hi_bgd_mon/ has expected formatting
- Confirm a generation of a new report matches style and content of previous reports
- Obsid made with prime: https://cxc.cfa.harvard.edu/mta/ASPECT/aca_hi_bgd_mon/events/obs_65524/index.html
- Confirm email notification on new event works (65524, 27689)
Status: Test complete. One oddity was that obsid 65524 did not show up in standard processing with a high background anomaly. The new outputs look correct and this seems benign, and likely indicates a problem with the source telemetry availability at the time. Will keep an eye on this but consider the prime release good to be promoted as flight.
Setup:
rm /proj/sot/ska3/test/data/arc3
# Made a new /proj/sot/ska3/prime/data/arc3 directory
mkdir /proj/sot/ska3/test/data/arc3
# Copied the /proj/sot/ska/data/arc3 contents into that directory
rsync -aruvz /proj/sot/ska/data/arc3/* /proj/sot/ska3/test/data/arc3/
mkdir -p /proj/sot/ska/www/ASPECT/arc3_prime_test/
mkdir -p /proj/sot/ska3/test/www/ASPECT/
ln -s /proj/sot/ska/www/ASPECT/arc3_prime_test/ /proj/sot/ska3/shiny/www/ASPECT/arc3
cd ~/git/arc
# as aca user
source /proj/sot/ska3/test/bin/ska_envs.csh
make install
Then ran the task pieces by hand to look for errors at the console:
source /proj/sot/ska3/test/bin/ska_envs.sh
jeanconn-fido> /proj/sot/ska3/test/share/arc3/get_iFOT_events.pl
jeanconn-fido> /proj/sot/ska3/test/share/arc3/get_web_content.pl
Warning: 500 Can't connect to space.umd.edu:443 (SSL connect attempt failed error:0A000126:SSL routines::unexpected eof while reading) for web data solar_wind (https://space.umd.edu/pm/)
jeanconn-fido> /proj/sot/ska3/test/share/arc3/get_web_content.pl
jeanconn-fido> /proj/sot/ska3/test/share/arc3/get_goes_x.py --h5=/proj/sot/ska3/test/data/arc3/GOES_X.h5
Warning: Data gap or error in X-ray data. Fetching 7-day JSON file
jeanconn-fido> /proj/sot/ska3/test/share/arc3/get_goes_x.py --h5=/proj/sot/ska3/test/data/arc3/GOES_X.h5
jeanconn-fido> /proj/sot/ska3/test/share/arc3/plot_goes_x.py --h5=/proj/sot/ska3/test/data/arc3/GOES_X.h5 --out=/proj/sot/ska3/test/www/ASPECT/arc3/goes_x.png
jeanconn-fido> /proj/sot/ska3/test/share/arc3/get_ace.py --h5=/proj/sot/ska3/test/data/arc3/ACE.h5
jeanconn-fido> /proj/sot/ska3/test/share/arc3/get_hrc.py --h5=/proj/sot/ska3/test/data/arc3/hrc_shield.h5 --data-dir=/proj/sot/ska3/test/data/arc3
Warning: Data gap or error in GOES proton data. Fetching 7-day JSON file
jeanconn-fido> /proj/sot/ska3/test/share/arc3/get_hrc.py --h5=/proj/sot/ska3/test/data/arc3/hrc_shield.h5 --data-dir=/proj/sot/ska3/test/data/arc3
jeanconn-fido> /proj/sot/ska3/test/share/arc3/plot_hrc.py --h5=/proj/sot/ska3/test/data/arc3/hrc_shield.h5 --out=/proj/sot/ska3/test/www/ASPECT/arc3/hrc_shield.png
jeanconn-fido> /proj/sot/ska3/test/share/arc3/make_timeline.py --data-dir=/proj/sot/ska3/test/data/arc3
jeanconn-fido> /proj/sot/ska3/test/share/arc3/make_timeline.py:36: MatplotlibDeprecationWarning: mplDeprecation was deprecated in Matplotlib 3.6 and will be removed two minor releases later. Use matplotlib.MatplotlibDeprecationWarning instead.
category=matplotlib.cbook.mplDeprecation
jeanconn-fido> /proj/sot/ska3/test/share/arc3/arc.pl
jeanconn-fido> /proj/sot/ska3/test/share/arc3/arc.pl -config arc3:arc_ops
jeanconn-fido> /proj/sot/ska3/test/share/arc3/arc_time_machine.pl
Examined output at https://cxc.cfa.harvard.edu/mta/ASPECT/arc3_prime_test/ and checked:
- HTML has right dates
- snapshot content appears correct
- plots match flight bye eye (the GOES data got a little stale when I ran a few parts of the test additional times)
On-the-side testing:
# remove symlink
rm /proj/sot/ska3/test/data/attitude_error_mon
# make test directory
mkdir /proj/sot/ska3/test/data/attitude_error_mon
# seed with flight data
cp -Ruva /proj/sot/ska/data/attitude_error_mon/* /proj/sot/ska3/test/data/attitude_error_mon
Install and test as aca user
aca-fido% source /proj/sot/ska3/test/bin/ska_envs.csh
aca-fido% cd ~/git/attitude_error_mon
aca-fido% make install
aca-fido% python /proj/sot/ska3/test/share/attitude_error_mon/att_err_mon.py --outdir ${SKA}/www/ASPECT/attitude_error_mon --datadir ${SKA}/data/attitude_error_mon
aca-fido% mkdir /proj/sot/ska/www/ASPECT/attitude_error_mon_prime
aca-fido% rsync -aruvz /proj/sot/ska3/test/www/ASPECT/attitude_error_mon/* /proj/sot/ska/www/ASPECT/attitude_error_mon_prime/
Test at command line for no errors
Works to make reasonable output:
- script runs without error
- plots are reasonable at cxc.harvard.when compared to current "flight" plots
-
cheta_syncfrom ska3-prime ran with no errors [TA, 2023-01-31]
Regression and unit tests provide full coverage of package user functionality and daily cron updates.
Regression and unit tests provide full coverage of package user functionality and daily cron updates.
- skawatch.py and hourly_watch.py run without error in the shiny environment.
- the html outputs look reasonable to visual inspection
- the tests in the repository appear out-of-date, and some fail (and fail the same ways in ska3/flight).
OK!
- Not critical, forward and fix.