Skip to content

Major Ska3 Update Testing Template

Jump to bottom Edit New page
Javier Gonzalez edited this page Mar 4, 2024 · 1 revision

Summary

This document describes testing plans and results for the transition to {version}, with Python {python_version} and updates to most of the core packages such as NumPy and Astropy.

Because of the significant and sweeping changes in the core packages that comprise Ska3, teams that use Ska3 as part of flight operations are responsible for the following:

  • Performing an audit to identify critical tools or software that depend on Ska3
  • Testing those tools using the Ska3 prime environment
  • Reporting results by directly editing this page, either with the results in full or by inserting a link to a separate test results page.

Forward and Fix

It is worth noting that many tools are not flight critical and being broken for a week or two is tolerable. In those cases the most efficient path may be "forward and fix". In other words, plan to wait until the promotion of Ska3 shiny to flight and then test at that time. This can be a useful strategy for tools where it is difficult to perform independent testing, e.g. if there are hardwired data paths.

Even in these cases it is important to identify in advance such tools along with a post-promotion test plan. "Testing" will likely consist of running the tool and evaluating the outputs for correctness.

Core functionality (Ska3 team responsibility)

Core testing is done via the Ska3 integration testing package which consists of running available package unit tests and regression test scripts using machinery in the testr package. This is done with the run_testr command, where some tests may be skipped on standalone platforms.

Current results for 2023.1rc12

See the test outputs are shown in:

Teams

ACA

  • Load review tools pass all tests (unit/regression)
  • sparkles/proseco pass all unit tests.
  • starcheck - ran full regression test set with /proj/sot/ska3/test with starcheck version 14.0.2 (in test and release).

FOT Mission Planning

  • FOT MP review and comments below

HRC ops

  • HRC team review and comments below

MTA

Is MTA still using a private version of SKA?

SOT MP

Ska3 cron processing

acdc

Date:

Run with

c3po-v$ cd ~/git/acdc
c3po-v$ mkdir -p regress_prime/VC2
# Get enough data to make the last two reports
c3po-v$ cp /dsops/GOT/input/2023*VC2* regress_prime/VC2
c3po-v$ cp /dsops/GOT/input/2022_3*VC2* regress_prime/VC2
c3po-v$ python -m acdc.process_vc2_to_l0 --vc2-root=regress_prime/VC2 --data-root=regress_prime
c3po-v$ python -m acdc.process_l0_to_quads --data-root=regress_shiny --start 2022:339
# I then edited regress_prime/vc2_index.dat to exclude early srdcs and ones before safe mode
# recovery as excluded.  The intent just to get 10 that line up with what was processed in flight
c3po-v$ python -m acdc.process_quads_to_cals --data-root=regress_prime
c3po-v$ python -m acdc.make_reports --data-root=regress_prime

Checks:

  • All processing runs successfully with reasonable outputs
  • Output report for 2023:023 cal looks reasonable
  • Output calibration image is consistent with flight (with note)

Note on comparing outputs:

The temperature values assigned to cals varied in this process by up to 0.08 C compared to flight. This is likely because flight processing is done soon after the cal and can be using available MAUDE telemetry from realtime. This will generate differences in temperatures and mask values because the temperatures are not available during the VC2 readout interval.

Diffs:

For the 2023:023 cal the diffs are:

  • Mean:
  • Stddev:
  • Max:
dat1 = fits.open("023/cal_2023_023_08_42_45.fits.gz")[0].data
dat2 = fits.open("023/flight_report/023/cal_2023_023_08_42_45.fits.gz")[0].data
diff = dat1 - dat2
argmax = np.argmax(np.abs(diff)).flatten()[0]
print(
    f"Mean diff: {np.mean(diff)}",
    f"Diff std dev: {np.std(diff)}",
    f"Max diff: {np.max(np.abs(diff))} ({dat1.ravel()[argmax]}, {dat2.ravel()[argmax]})",
)

acis_taco

Date:

Reprocess MAR2023 with Ska3-flight and {version}:

cd ~/git/acis_taco
mkdir tmp/acis_taco
cd cd tmp/acis_taco
mkdir flight
mkdir prime
/proj/sot/ska3/flight/bin/skare /proj/sot/ska3/flight/share/acis_taco/make_esaview_data.py --nweeks=1 --data-root=flight
/proj/sot/ska3/test/bin/skare /proj/sot/ska3/test/share/acis_taco/make_esaview_data.py --nweeks=1 --data-root=prime

Then from ipython

import pickle                                                                                                 
dat1 = pickle.load(open('tmp/acis_taco/flight/MAR2023.pkl', 'rb'))                                         
dat2 = pickle.load(open('tmp/acis_taco/prime/MAR2023.pkl', 'rb')) 
np.all(dat1['illums'] == dat2['illums'])  # True
np.all(dat1['times'] == dat2['times'])  # True

Result:

  • OK

aca_hi_bgd

Date:

Prime test plan:

Comments:

arc

Date:

Setup:

rm /proj/sot/ska3/test/data/arc3
# Made a new /proj/sot/ska3/prime/data/arc3 directory
mkdir /proj/sot/ska3/test/data/arc3
# Copied the /proj/sot/ska/data/arc3 contents into that directory
rsync -aruvz /proj/sot/ska/data/arc3/* /proj/sot/ska3/test/data/arc3/

mkdir -p /proj/sot/ska/www/ASPECT/arc3_prime_test/
mkdir -p /proj/sot/ska3/test/www/ASPECT/
ln -s /proj/sot/ska/www/ASPECT/arc3_prime_test/ /proj/sot/ska3/shiny/www/ASPECT/arc3

cd ~/git/arc
# as aca user
source /proj/sot/ska3/test/bin/ska_envs.csh
make install

Then ran the task pieces by hand to look for errors at the console:

source /proj/sot/ska3/test/bin/ska_envs.sh
jeanconn-fido> /proj/sot/ska3/test/share/arc3/get_iFOT_events.pl
jeanconn-fido> /proj/sot/ska3/test/share/arc3/get_web_content.pl
Warning: 500 Can't connect to space.umd.edu:443 (SSL connect attempt failed error:0A000126:SSL routines::unexpected eof while reading) for web data solar_wind (https://space.umd.edu/pm/)
jeanconn-fido> /proj/sot/ska3/test/share/arc3/get_web_content.pl
jeanconn-fido> /proj/sot/ska3/test/share/arc3/get_goes_x.py --h5=/proj/sot/ska3/test/data/arc3/GOES_X.h5
Warning: Data gap or error in X-ray data.  Fetching 7-day JSON file
jeanconn-fido> /proj/sot/ska3/test/share/arc3/get_goes_x.py --h5=/proj/sot/ska3/test/data/arc3/GOES_X.h5
jeanconn-fido> /proj/sot/ska3/test/share/arc3/plot_goes_x.py --h5=/proj/sot/ska3/test/data/arc3/GOES_X.h5 --out=/proj/sot/ska3/test/www/ASPECT/arc3/goes_x.png
jeanconn-fido> /proj/sot/ska3/test/share/arc3/get_ace.py --h5=/proj/sot/ska3/test/data/arc3/ACE.h5
jeanconn-fido> /proj/sot/ska3/test/share/arc3/get_hrc.py --h5=/proj/sot/ska3/test/data/arc3/hrc_shield.h5 --data-dir=/proj/sot/ska3/test/data/arc3
Warning: Data gap or error in GOES proton data.  Fetching 7-day JSON file
jeanconn-fido> /proj/sot/ska3/test/share/arc3/get_hrc.py --h5=/proj/sot/ska3/test/data/arc3/hrc_shield.h5 --data-dir=/proj/sot/ska3/test/data/arc3
jeanconn-fido> /proj/sot/ska3/test/share/arc3/plot_hrc.py --h5=/proj/sot/ska3/test/data/arc3/hrc_shield.h5 --out=/proj/sot/ska3/test/www/ASPECT/arc3/hrc_shield.png
jeanconn-fido> /proj/sot/ska3/test/share/arc3/make_timeline.py --data-dir=/proj/sot/ska3/test/data/arc3
jeanconn-fido> /proj/sot/ska3/test/share/arc3/make_timeline.py:36: MatplotlibDeprecationWarning: mplDeprecation was deprecated in Matplotlib 3.6 and will be removed two minor releases later. Use matplotlib.MatplotlibDeprecationWarning instead.
  category=matplotlib.cbook.mplDeprecation
jeanconn-fido> /proj/sot/ska3/test/share/arc3/arc.pl 
jeanconn-fido> /proj/sot/ska3/test/share/arc3/arc.pl -config arc3:arc_ops
jeanconn-fido> /proj/sot/ska3/test/share/arc3/arc_time_machine.pl

Examined output at https://cxc.cfa.harvard.edu/mta/ASPECT/arc3_prime_test/ and checked:

  • HTML has right dates
  • snapshot content appears correct
  • plots match flight bye eye (the GOES data got a little stale when I ran a few parts of the test additional times)

attitude_error_mon

Date:

On-the-side testing:

# remove symlink
rm /proj/sot/ska3/test/data/attitude_error_mon
# make test directory
mkdir /proj/sot/ska3/test/data/attitude_error_mon
# seed with flight data
cp -Ruva /proj/sot/ska/data/attitude_error_mon/* /proj/sot/ska3/test/data/attitude_error_mon

Install and test as aca user

aca-fido% source /proj/sot/ska3/test/bin/ska_envs.csh
aca-fido% cd ~/git/attitude_error_mon
aca-fido% make install
aca-fido% python /proj/sot/ska3/test/share/attitude_error_mon/att_err_mon.py --outdir ${SKA}/www/ASPECT/attitude_error_mon --datadir ${SKA}/data/attitude_error_mon
aca-fido% mkdir /proj/sot/ska/www/ASPECT/attitude_error_mon_prime
aca-fido% rsync -aruvz /proj/sot/ska3/test/www/ASPECT/attitude_error_mon/* /proj/sot/ska/www/ASPECT/attitude_error_mon_prime/

Test at command line for no errors

Works to make reasonable output:

  • script runs without error
  • plots are reasonable at cxc.harvard.when compared to current "flight" plots

cheta

Date:

  • cheta_sync from {version} ran with no errors.

Regression and unit tests provide full coverage of package user functionality and daily cron updates.

fss_check3

?

kadi

No extra tests. Regression and unit tests provide full coverage of package user functionality and daily cron updates.

arc5gl

Date:

  • confirm cmdline works fine in test environment to fetch files

mica

No extra tests. Regression and unit tests provide full coverage of package user functionality and daily cron updates.

perigee_health_plots

not tested. Can be updated quickly if issues pop up.

skawatch3

Date:

  • skawatch.py and hourly_watch.py run without error in the test environment.
  • html outputs look reasonable to visual inspection
  • OK

tests in the repository appear out-of-date, and some fail (and fail the same ways in ska3/flight).

twiki-wg

  • Not critical, forward and fix.
Add a custom footer

For Users

For Maintainers

Continuous Integration

Old Pages

Clone this wiki locally