Release ska3-flight 2023.1

[javierggt]

This is a major release with updates to all packages in ska3-core. The python target version is 3.10.

Package and Test

• Create branch named {version}-branch
• Create PR titled {version} from {version}-branch into master
• Create a pre-release {version}rc{N} at the latest commit in the branch
• Check packages are built (for prime, packages are built using the workflow in sot/test-actions).
• Check that all packages are in prime channel for the three platforms.
• Notify FOT of new version
• Check that tests pass in all platforms.
• List in PR any data products that need to be promoted to $SKA/data (check PR's interface impacts).
• Fix warnings in ska3-prime. #1054
• Functional tests pass.
• Install on HEAD and GRETA Linux test environments (ska3/test).
• Confirm that test data (SKA/data) is appropriate for release testing (any custom test data is set or previous test data cleaned out)
• Run testr on HEAD and GRETA
• Add related issues to PR (skare3-milestone-issues ...).
• Document all changes (skare3-changes-summary ...).
• Write summary + highlight relevant changes.
• Document test results (skare3-test-dashboard ...).
• Create FSDS Jira ticket and wait for approval.

Promote:

• Promote packages to flight conda channel (skare3-promote ...).
• Announce to aca@cfa/slack.
• Update ska3/flight on HEAD as aca user (see below).
• Update ska3/flight on GRETA as SOT user (see below).
• Promote any data products listed in PR to be promoted.
• Run ska_testr on HEAD and GRETA.
• Document test results (skare3-test-dashboard ...).
• Announce to aca@cfa/slack.
• Announce to sot@cfa and fot@ipa.
• Merge PR.

Post-promotion tasks

• Update the cheta task schedule cron job to be: (ska3) skare task_schedule3.pl -package cheta. The current cron job spec is looking for task_schedule.cfg in share/eng_archive.
• Update install directions for users wiki page
• Update github runners keep-alive script

Test Installation steps:

These are the steps to follow when creating the test environment.

mv /proj/sot/ska3/test /proj/sot/ska3/test-shiny
wget https://repo.anaconda.com/miniconda/Miniconda3-py310_23.1.0-1-Linux-x86_64.sh
bash ~/Miniconda3-py310_23.1.0-1-Linux-x86_64.sh -b -p /proj/sot/ska3/test

log out and back in

eval "$(/proj/sot/ska3/test/bin/conda shell.bash hook)"

create condarc with (note that the flight channel is not in it because it is not compatible):

channels:
  - https://icxc.cfa.harvard.edu/aspect/ska3-conda/test

conda install mamba
mamba install ska3-core==2023.1rc12
mamba install ska3-flight==2023.1rc12
mamba install ska3-perl==2023.1rc12

mkdir /proj/sot/ska3/test/data
cd /proj/sot/ska3/test/data
ln -s /proj/sot/ska/data/* .

Flight Installation steps:

These are the steps to follow when creating the flight environment.

1. Set a calendar reminder to remove archived version of ska3/flight (flight-2022.13)
2. Promote conda channel from test to flight
3. /proj/sot/ska3/flight HEAD and GRETA deployment
4. Fallback

Promote conda channel from test to flight

• Pause cron jobs that rsync conda channels (jgonzale and aca users)
• Move flight channel to flight-shiny, create new flight channel, and promote. On HEAD as aca user:

  cd /proj/sot/ska/www/ASPECT/ska3-conda
  mv flight flight-shiny
  mkdir flight
  cd /proj/sot/ska/www/ASPECT_ICXC/ska3-conda
  mv flight flight-shiny
  mkdir flight
  source /proj/sot/ska3/flight/bin/ska_envs.[c]sh
  conda index flight
  skare3-promote ...
  

  at this point it is possible that some packages need to be copied by hand from flight-shiny into flight.

HEAD deployment

• Schedule downtime and notify users of ska3/flight

• Pause all cron jobs running out of ska3/flight as aca user

  touch /proj/sot/ska/data/task_schedule/master_heart_attack 
  

• Pause Replan Central which has a heart attack override

  cd /proj/sot/ska/data/arc
  touch task_sched_heart_attack
  

  Note that we are still likely to get cron task warnings, as the cron tasks are going to be trying to run ska3/flight/skare which will be missing. I'm not actually sure that the heart attacks are going to help, but it seems more controlled.

• As aca user, move ska3/flight

  cd /proj/sot/ska3
  mv flight flight-2022.13
  

• As aca user, install a clean Miniconda into ska3/flight

  wget https://repo.anaconda.com/miniconda/Miniconda3-py310_23.1.0-1-Linux-x86_64.sh
  bash Miniconda3-py310_23.1.0-1-Linux-x86_64.sh -b -p /proj/sot/ska3/flight
  

• log out and back in

• As aca user, copy condarc from /proj/sot/ska3/flight-2022.13:

  cp /proj/sot/ska3/flight-2022.13/condarc /proj/sot/ska3/flight/
  

  the contents of the file should be:

  channels:
   - https://icxc.cfa.harvard.edu/aspect/ska3-conda/flight
  

• As aca user, install the ska3 packages into the root environment, confirming that ska3-flight 2023.1 is installed by default. This did not worked:

  eval "$(/proj/sot/ska3/flight/bin/conda shell.bash hook)"
  conda install ska3-flight ska3-perl
  

  did this instead:

  eval "$(/proj/sot/ska3/flight/bin/conda shell.bash hook)"
  conda install mamba 
  mamba install ska3-flight ska3-perl
  

• As aca user, set up data and www links

  cd /proj/sot/ska3/flight
  ln -s /proj/sot/ska/data ./
  ln -s /proj/sot/ska/www ./
  

• Quick sanity check use cheta and matplotlib

  ipython --matplotlib
  from cheta import fetch
  dat = fetch.Msid('tephin', '2020:001', '2020:030')
  dat.plot()
  

  I logged in using ssh -Y from my mac, and saw the following error but it worked:

  libGL error: No matching fbConfigs or visuals found
  libGL error: failed to load driver: swrast
  

• As aca user, manually install ska3 packages that are installed via Makefile:

  • arc: 4.7.2
  • attitude_error_mon: 0.1.0
  • twiki-wg: 0.2.0
  • fss_check: 0.2.1
  • obc_rate_noise_plots: 0.1.0

  In each case get into the git repository, check out the indicated release, and
  and then install with make install as aca user.

• Run post-install Ska integration tests run_testr (useful given the full install; preferably run as an unprivileged user):

• Restart cron jobs by removing the master heart attack and job heart attack files.

  rm /proj/sot/ska/data/task_schedule/master_heart_attack
  rm /proj/sot/ska/data/arc/task_sched_heart_attack
  

GRETA deployment

• Schedule downtime and notify users of ska3/flight on GRETA
• Stop ska sync cron task by commenting out the line in the crontab.
• As SOT user move /proj/sot/ska3/flight

  cd /proj/sot/ska3
  mv flight flight-2022.13
  

• As SOT user on chimchim or cheru, install a clean Miniconda into ska3/flight

  rsync ccosmos.cfa.harvard.edu:/proj/sot/ska/www/ASPECT_ICXC/ska3-conda/Miniconda3-py310_23.1.0-1-Linux-x86_64.sh .
  bash Miniconda3-py310_23.1.0-1-Linux-x86_64.sh -b -p /proj/sot/ska3/flight  
  

• As SOT user, copy condarc from /proj/sot/ska3/flight-2022.13:

  cp /proj/sot/ska3/flight-2022.13/condarc /proj/sot/ska3/flight/
  

• As SOT user, install the ska3 packages into the root environment, confirming that ska3-flight 2023.1 is installed by default

  eval "$(/proj/sot/ska3/flight/bin/conda shell.bash hook)"
  conda install mamba
  mamba install ska3-flight ska3-perl
  

• As SOT user, set up data link

  cd /proj/sot/ska3/flight
  ln -s /proj/sot/ska/data ./
  

• As SOT user, install sync_ska_data_occ
• Quick sanity check use cheta and matplotlib

  ipython --matplotlib
  from cheta import fetch
  dat = fetch.Msid('tephin', '2020:001', '2020:030')
  dat.plot()
  

• Run post-install Ska integration tests, run_testr, as a non-privileged user (useful given the full install)
• Restart sync job

Fallback

In the unlikely event of requiring a fallback to the current flight version 2022.13
(using Python 3.8), this can be done easily on HEAD and GRETA with:

cd /proj/sot/ska3
mv flight flight-2023.1
mv flight-2023.13 flight

[jeanconn]

Regarding testing, https://github.com/sot/skare3/wiki/Ska3-shiny-testing document is a good starting point.