Ska3 runtime environment for users

The Ska3 runtime environment consists of an integrated suite of Python-3 packages and associated applications. The installation and update of these packages is done using the Anaconda Python distribution and the Conda package manager. This allows simple installation on linux machines or Mac laptops. Similar support for Windows is under consideration.

Maintaining Ska3 runtime environment
Installing Ska3 runtime environment
Configuration hints
Testing Ska3
Ska Data
Installing and maintaining Ska3 development (masters) environment

For discussion about motivation for Python 3, new features, and converting from Python 2.7 to Python 3, see:

Maintaining Ska3 runtime environment

If you have a Ska3 runtime environment already installed, maintaining your environment is straightforward. When there is a new release (or just periodically) do:

mamba update ska3-flight --override-channels -c https://icxc.cfa.harvard.edu/aspect/ska3-conda/flight

Installing Ska3 runtime environment

If you are using Ska3 on the HEAD or GRETA linux networks or via MATLAB tools, then there is no need to install Ska3.

However, you may wish to look through the [Configuration hints][#configuration-hints] section.

Power users note: on the HEAD linux network there can be a substantial improvement in package import times if you install and maintain your own Ska3 installation on a local disk. The production installation on /proj/sot is served via a remote NetApp which can be quite slow.

Installing conda on linux CentOS-7+, MacOSX, or Windows 10

The basis for installing, managing, and using Ska3 is the Conda package manager and this needs to be installed before anything else can be done. If you already have the conda package manager installed with version 4.8 or higher (conda --version), then you can skip these steps and go to the Create Ska3 environment section.

Important note about older conda

If you have a version of conda older than 4.8, this is not compatible with ska3. We recommend deleting that old conda and the associated environment.

The steps for creating a new conda environment are as follows:

Get one of the following files by URL:

https://repo.continuum.io/miniconda/Miniconda3-py310_23.1.0-1-Linux-x86_64.sh
https://repo.continuum.io/miniconda/Miniconda3-py310_23.1.0-1-MacOSX-x86_64.sh
https://repo.continuum.io/miniconda/Miniconda3-py310_23.1.0-1-Windows-x86_64.exe

For Windows follow the instructions for Installing on Windows or consult your local Windows expert.

For linux and MacOSX proceed with the following steps.

Open a terminal window and change to the downloads directory for your browser.
Ensure that you have git installed in your system by typing git --version on the command line. If not, install it via https://git-scm.com/downloads.
The installer is called something like Miniconda-py310_23.1.0-1-<OS>.sh. Find it and then enter the following (without the %, which is used as the linux prompt in all these examples):

bash Miniconda3-py310_23.1.0-1-MacOSX-x86_64.sh

Hit ENTER, then agree to the license terms, and finally specify the installation directory. This will be created by the installer and needs to be a directory that doesn t currently exist. A fine choice is the default of ~/miniconda3 (which is what is assumed for the rest of the installation), but you may want to install to a different directory depending on the details of your system.
At the end it asks about updating your .bashrc or .bash-profile or .cshrc file. We recommend accepting (say yes).
Now create a new terminal window.

This gives you a minimal working Python along with the package manager utility conda. Check hat this is working by doing:

which python
--> shows the expected path, e.g. $HOME/miniconda3/bin/python

Create Ska3 environment

Now you can install all the Ska3 packages and required dependencies to an environment named ska3 by copy-pasting this long command:

conda create -n ska3 ska3-flight --override-channels -c https://icxc.cfa.harvard.edu/aspect/ska3-conda/flight

On linux and MacOSX you can optionally install ska3-perl as well. This is mostly for aspect-specific apps like starcheck.

After you see the list of packages, accept, and see that the install finished. Then you can get into the Ska3 environment

conda activate ska3

It's done and now you have a ska3 flight environment! The name ska3 is not magic, and you could have called it my-ska or whatever you want.

If you make a new shell window then you need to conda activate ska3 to get into Ska3 again.

NOTE: sometimes, the conda create... command above fails with a long message about conflicts. If that is the case, try the following instead:

conda create -n ska3 ska3-flight
conda activate ska3-flight
conda install --override-channels -c https://icxc.cfa.harvard.edu/aspect/ska3-conda/flight mamba
mamba install --override-channels -c https://icxc.cfa.harvard.edu/aspect/ska3-conda/flight ska3-flight

Configuration hints

Linux alias for production Ska3

It is useful to define aliases to get into the Ska3 production environment on HEAD or GRETA and adjust your prompt to indicate that you are using Ska3. The command and file to modify depends on the shell you are using and the network. First if you don't know what shell you are using then do:

  echo $0

This should say either csh or tcsh or bash.

============= =============  =============
Shell             Network        File
============= =============  =============
csh or tcsh   HEAD           ~/.cshrc.user
csh or tcsh   GRETA          ~/.cshrc
bash          HEAD or GRETA  ~/.bashrc
============= =============  =============

Now put the appropriate line at the end of the indicated file:

  # Csh or tcsh
  alias ska3  'source /proj/sot/ska3/flight/bin/ska_envs.csh; set prompt="ska3-$prompt:q"'

  # Bash
  alias ska3='source /proj/sot/ska3/flight/bin/ska_envs.sh; export PS1="ska3-$PS1"'

The ska3 alias is a one-way ticket. Once you get into the Ska3 environment the recommended way to get out or change to a different version is to start a new window.

IPython startup file

The interactive Python tool ipython has a nice feature that you can define custom "magic" commands to perform actions with a single command. One useful thing to do is to import a number of common Ska3 packages with a single command %impska at the ipython command line. To do this, follow the instructions below.

First locate your default IPython profile directory with:

ipython profile locate

In that directory, if there is not already a sub-directory startup then create it now.

Finally, in the startup sub-directory within your default IPython profile directory make a new file called 10_import_ska3.py and copy/paste the following into that file. You can easily customize yourself following the obvious pattern.

from IPython.core.magic import Magics, magics_class, line_magic
from IPython import get_ipython

ip = get_ipython()


@magics_class
class MyMagics(Magics):
    @line_magic
    def impska(self, line):
        ip.ex('import cheta.fetch_eng as fetch')
        ip.ex('from cheta import fetch_sci')
        ip.ex('from Chandra.Time import DateTime')
        ip.ex('from cxotime import CxoTime')
        ip.ex('from Ska.Matplotlib import plot_cxctime, cxctime2plotdate')
        ip.ex('from kadi import events')
        ip.ex('from astropy.table import Table')
        ip.ex('import numpy as np')
        ip.ex('import matplotlib.pyplot as plt')
        ip.ex('from Ska.tdb import msids as TDB')
        ip.ex('import kadi')
        ip.ex('import cxotime')
        ip.ex('print(f"cheta version={fetch.__version__}")')
        ip.ex('print(f"kadi version={kadi.__version__}")')
        ip.ex('print(f"cxotime version={cxotime.__version__}")')


ip.register_magics(MyMagics)

Now start up ipython and enter %impska. You should see something like this:

(ska3) ➜  ~ ipython
Python 3.8.3 (default, Jul  2 2020, 11:26:31) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: %impska
cheta version=4.50.1
kadi version=5.4.0
cxotime version=3.2.3

Testing Ska3

Congratulations, you should now have all the Ska3 packages installed!

Quick check

As a quick smoke test that things are working, confirm that the following works and makes a line plot:

ipython --matplotlib
>>> import matplotlib.pyplot as plt
>>> from Chandra.Time import DateTime
>>> import ska_path
>>>
>>> plt.plot([1, 2])
>>> DateTime(100).date
'1998:001:00:00:36.816'

Comprehensive check

A comprehensive check that everything is working can be done by running the full Ska3 integration test suite. This will take up to a half hour. Before starting this you must ensure that the standard Ska data are installed.

# Change to a directory for git repos, or a temporary directory
cd ~/git  # Or whatever
git clone https://github.com/sot/ska_testr.git
cd ska_testr
run_testr --test-spec test_spec_standalone

This will provide a summary of test results at the end, with detailed logs available in the outputs/logs directory.

Ska data

To make full use of the Ska environment you need various package data, e.g.:

Ska engineering archive
Kadi databases
Commanded states databases
AGASC star catalog
Telemetry database (MSIDs)

The very first thing to do is define the disk location where you want to store the Ska data. Packages and applications in the Ska runtime environment use an environment variable SKA to define the root directory of data and other shared resources. On the HEAD or GRETA networks this root is the familiar /proj/sot/ska3/flight.

For a machine (e.g. your laptop) not on the HEAD or GRETA networks you need to make a directory that will hold the Ska data. A reasonable choice is putting this in a ska directory in your home directory:

mkdir ~/ska
setenv SKA ${HOME}/ska  # csh / tcsh
export SKA=${HOME}/ska  # bash

If you have made a local Ska environment on a machine on the HEAD or GRETA networks, you can just set the SKA environment variable to point at the existing root:

setenv SKA /proj/sot/ska  # csh / tcsh
export SKA=/proj/sot/ska  # bash

Syncing data

For a machine not on the HEAD or GRETA networks you need to get at least a subset of the data on your machine. FOT users may want to start up MATLAB which will run a task to update the Ska data used by those tools. Other users or FOT users with different data needs will use a script called ska_sync that is installed as part of Ska. This assume you have set the SKA environment variable and created a directory as shown above.

The first thing is to try running it and getting the online help:

% ska_sync --help
usage: ska_sync [-h] [--user USER] [--install] [--force]

Synchronize data files for Ska runtime environment

Arguments
=========

optional arguments:
  -h, --help   show this help message and exit
  --user USER  User name for remote host
  --install    Install sync config file in default location
  --force      Force overwrite of sync config file

Next you need to install a copy of the template configuration file into your Ska root directory. This will let you customize which data you want and how to get it. This installation is done with:

% echo $SKA  # Check the SKA root directory
/Users/aldcroft/ska

% ska_sync --install
Wrote ska sync config file to /Users/aldcroft/ska/ska_sync_config

Now you need to edit the file it just created and set the remote host for getting data and the remote host user name. Choose either kadi.cfa.harvard.edu (HEAD) or the IP address for cheru (OCC) and edit accordingly. Likewise put in the corresponding user name:

# Host machine to supply Ska data (could also be chimchim but kadi works
# from OCC VPN just as well).
host: kadi.cfa.harvard.edu

# Remote host user name.  Default is local user name.
# user: name

If you want Ska engineering archive access then following the instructions in Syncing Ska engineering archive data

Finally do the sync step:

% ska_sync
Loaded config from /Users/aldcroft/ska/ska_sync_config

This prints something like below:

COPY and PASTE the following at your terminal command line prompt:

  rsync -arzv --progress --size-only --files-from="/Users/aldcroft/ska/ska_sync_files" \
    aldcroft@kadi.cfa.harvard.edu:/proj/sot/ska/ "/Users/aldcroft/ska/"

As instructed copy and paste the rsync line, after which you will need to enter your password. At this point it will sync the relevant Ska data files into your local Ska root.

NOTE for Windows users (but not on FOT laptops): one possibility for running the rsync command is to use Windows Subsystem for Linux. This has been successfully demonstrated for a standalone Windows machine.

As long as you don t change your config file, you can just re-run that same command to re-sync as needed.

To test that the data are really there make sure you can reproduce the following:

% ipython --matplotlib
>>> from Ska.tdb import msids
>>> msids.find('tephin')
[<MsidView msid="TEPHIN" technical_name="EPHIN SENSOR HOUSING TEMP">]

>>> from kadi import events
>>> events.normal_suns.filter('2014:001')
<NormalSun: start=2014:207:07:04:09.331 dur=65207>

>>> from Chandra.cmd_states import fetch_states
>>> fetch_states('2011:100', '2011:101', vals=['obsid'])
    ...
    SOME WARNINGS WHICH ARE OK and will get patched up later
    ...
[ ('2011:100:11:53:12.378', '2011:101:00:26:01.434', 418823658.562, 418868827.618, 13255)
  ('2011:101:00:26:01.434', '2011:102:13:39:07.421', 418868827.618, 419002813.605, 12878)]

Syncing Ska engineering archive data

Syncing Ska engineering data to your standalone installation (laptop) is feasible. Circa mid-2019, the archive about 170 Gb. Copying the entire archive over VPN network will take a while, but it has been done.

Keeping the archive up to date should be done using cheta_sync, which is documented in the Cheta telemetry archive tutorial.

Mission planning backstop and OR-lists

rsync -av --prune-empty-dirs --exclude='/*/*/*/' --include='*/' \
 --include='???????/ofls?/*.backstop' --include='???????/ofls?/*.or' --exclude='*' \
 kady:/data/mpcrit1/mplogs/2019/ $SKA/data/mpcrit1/mplogs/2019/

Installing and maintaining Ska3 development (masters) environment

Users that want to work in an environment that contains the latest development version (the master branch) of all Ska3 packages can do so as follows. This takes advantage of a distribution channel named masters that has the latest conda build for the master branch of all Ska3 packages.

Installing

The command below will select packages from masters where the version is greater than the release version.

# Mac / linux (note: ska3-perl-latest is optional)
conda create -n ska3-dev --override-channels \
  -c https://icxc.cfa.harvard.edu/aspect/ska3-conda/masters \
  -c https://icxc.cfa.harvard.edu/aspect/ska3-conda/flight \
  ska3-core-latest ska3-flight-latest ska3-perl-latest

# Windows
conda create -n ska3-dev --override-channels `
  -c https://icxc.cfa.harvard.edu/aspect/ska3-conda/masters `
  -c https://icxc.cfa.harvard.edu/aspect/ska3-conda/flight `
  ska3-core-latest ska3-flight-latest

Updating a single package with conda

For a Ska3-dev environment you can easily update a single Ska3 package. For instance if you wanted to revert to version 3.1.0 of dea_check (since master of dea_check is ahead of that), you would do the following:

conda install -c $ska3conda/flight --override-channels dea_check=3.1.0

The commands below to update all packages to the latest would undo this.

Getting a bit braver, if you wanted to install numpy 1.19 then this would work:

conda install --override-channels \
  -c https://icxc.cfa.harvard.edu/aspect/ska3-conda/flight -c defaults \
  numpy=1.19

Updating a single package with pip

If you have your own development version of a package and you want to install it into your dev environment, this is easily done with pip:

pip install .

This will override the existing conda package (if it exists). Doing a conda list will now show the package as being pip-installed and will show the appropriate dev version.

Updating all packages to the latest masters

You can update to the latest version in the master branch for all Ska3 packages with the following:

# Mac / linux
mamba update --update-deps --override-channels \
  -c https://icxc.cfa.harvard.edu/aspect/ska3-conda/masters \
  ska3-flight-latest

# Windows
conda create -n ska3-dev --override-channels `
  -c https://icxc.cfa.harvard.edu/aspect/ska3-conda/masters `
  ska3-flight-latest

Provide feedback

Saved searches

Use saved searches to filter your results more quickly