diff --git a/install/README.md b/install/README.md index 3d05ae7..53f3bec 100644 --- a/install/README.md +++ b/install/README.md @@ -6,6 +6,7 @@ instrument. **Contents** - [Installation](#installation) + - [Typical Directory Layout](#typical-directory-layout) - [Activate Conda "base" Environment](#activate-conda-base-environment) - [Install MongoDB server](#install-mongodb-server) - [Install Databroker Configuration File](#install-databroker-configuration-file) @@ -14,6 +15,13 @@ instrument. - [Test that Bluesky Works](#test-that-bluesky-works) - [Create IPython Profile for Bluesky](#create-ipython-profile-for-bluesky) - [Install Instrument Package](#install-instrument-package) + - [Download the Starting Point](#download-the-starting-point) + - [Edit Basic Details](#edit-basic-details) + - [conda environment name](#conda-environment-name) + - [databroker catalog name](#databroker-catalog-name) + - [instrument-based metadata](#instrument-based-metadata) + - [Remove training repository components](#remove-training-repository-components) + - [Install](#install) - [Commit Instrument Package to Version Control](#commit-instrument-package-to-version-control) - [Translate Previous SPEC Configuration](#translate-previous-spec-configuration) - [Add Environment Configuration to .bash_aliases](#add-environment-configuration-to-bash_aliases) @@ -31,6 +39,30 @@ commands, check that you are using the `bash` shell first. Here's some [advice](https://stackoverflow.com/questions/3327013/how-to-determine-the-current-shell-im-working-on) to determine the current shell. +## Typical Directory Layout + +```text +~/ + bin/ + blueskyStarter.sh --> ~/bluesky/blueskyStarter.sh + bluesky/ + blueskyStarter.sh + console/ + __start_bluesky_instrument__.py + instrument/ + .ipython-bluesky/ + profile_bluesky/ + startup/ + __start_bluesky_instrument__.py --> ~/bluesky/console/__start_bluesky_instrument__.py +``` + +and [`__start_bluesky_instrument__.py`](https://github.com/BCDA-APS/bluesky_training/blob/main/bluesky/console/__start_bluesky_instrument__.py) is summarized by: + +```py +import pathlib, sys +sys.path.append(str(pathlib.Path.home() / "bluesky")) +from instrument.collection import * +``` ## Activate Conda "base" Environment @@ -64,7 +96,6 @@ mintadmin@mint-vm:/tmp$ conda env list # conda environments: # base * /home/mintadmin/Apps/anaconda - ``` @@ -73,8 +104,8 @@ base * /home/mintadmin/Apps/anaconda If you need to provide your own MongoDB server, follow this [guide](https://github.com/BCDA-APS/use_bluesky/wiki/mongodb-server). -At the APS, the beam line controls group (BCDA) will assign the -REPOSITORY and its SERVER. +At the APS, the IT group manages the MongoDB servers and the beam line +controls group (BCDA) will make assignments (REPOSITORY and SERVER). ## Install Databroker Configuration File @@ -83,7 +114,7 @@ databases) on the MongoDB database server. Follow this template: -``` +```yml sources: REPOSITORY: args: @@ -95,30 +126,35 @@ sources: Replace the `REPOSITORY` and `SERVER` terms and write to file: `~/.local/share/intake/catalogs.yml`. +NOTE: The actual directory is `${XDG_CONFIG_HOME}/intake/catalogs.yml`. The example above assumes the default value `XDG_CONFIG_HOME=$HOME}/.local/share`. + At the APS, the beam line controls group (BCDA) will assign the -REPOSITORY and its SERVER. +REPOSITORY and its SERVER. The maintain a list of all +[APS configurations](https://git.aps.anl.gov/bcda/bluesky-catalogs/-/blob/master/APS_assignments.yml) +(restricted access). ## Install Bluesky Environment Get the installation configuration ([YAML](https://yaml.org)) file. The -file is versioned based on the APS run cycle. This YAML file is for the -first run (January-April) in 2021. +file is versioned based on the APS run cycle. The YAML file shown here is for the +third run (October-December) in 2022. Use the most recent YAML file available +in the repository. The appropriate environment name is built into the file. -``` +```bash cd /tmp -wget https://github.com/BCDA-APS/use_bluesky/raw/main/install/environment_2022_2.yml +wget https://github.com/BCDA-APS/use_bluesky/raw/main/install/environment_2022_3.yml ``` -Create the custom conda environment named in the file (`bluesky_2021_1`): +Create the custom conda environment named in the file (`bluesky_2022_3`): -``` -conda env create -f environment_2021_1.yml +```bash +conda env create -f environment_2022_3.yml ``` ## Activate Bluesky Environment -``` -conda activate bluesky_2021_1 +```bash +conda activate bluesky_2022_3 ``` ### Test that Bluesky Works @@ -335,45 +371,113 @@ If there is an existing `~/.ipython` directory (perhaps created for other use from this account), then choose a unique directory for bluesky. Typical alternative is `~/.ipython-bluesky`. -``` -export BLUESKY_DIR=~/.ipython +```bash +export BLUESKY_DIR=~/.ipython-bluesky +mkdir -p "${BLUESKY_DIR}" ipython profile create bluesky --ipython-dir="${BLUESKY_DIR}" ``` - ## Install Instrument Package -Remove the existing `startup` directory (created from `ipython profile -create` step above): +### Download the Starting Point +These steps install the instrument package from the training +repository. It is the best starting point for a new installation. + +```bash +cd /tmp +git clone https://github.com/BCDA-APS/bluesky_training +cd bluesky_training/bluesky ``` -cd "${BLUESKY_DIR}/profile_bluesky" -rm startup/README -rmdir startup + +### Edit Basic Details + +Edit content in the bluesky directory (and subdirectories below). + +#### conda environment name + +```bash +sed -i s+"training_2022"+"bluesky_2022_3"+g ./blueskyStarter.sh +sed -i s+"training_2022"+"bluesky_2022_3"+g ./run_qstarter_py.sh +``` + +#### databroker catalog name + +Use the same name as `REPOSITORY` in the +[Install Databroker Configuration File](#install-databroker-configuration-file) +section above. + +```bash +CATALOG_NAME=USE_ASSIGNED_CATALOG_NAME +sed -i s+"training"+"${CATALOG_NAME}"+g console/__start_bluesky_instrument__.py +sed -i s+"training"+"${CATALOG_NAME}"+g export_data.sh +sed -i s+"training"+"${CATALOG_NAME}"+g instrument/framework/initialize.py +sed -i s+"databroker_catalog training"+"databroker_catalog ${CATALOG_NAME}"+g instrument/iconfig.yml +sed -i s+"training"+"${CATALOG_NAME}"+g qserver.sh +sed -i s+"training"+"${CATALOG_NAME}"+g qstarter.py +sed -i s+"training"+"${CATALOG_NAME}"+g run_qstarter_py.sh ``` -Download the install script: +#### instrument-based metadata + +Start with some basic metadata about the instrument. +Leave a placeholder to record a user's proposal ID. +Keep in mind that metadata keys, such as these, may be used +to find and retireve data from the databroker. +```bash +# no white space in NAME_THE_INSTRUMENT +sed -i s+"beamline_id: Bluesky_training"+"beamline_id: NAME_THE_INSTRUMENT"+g instrument/iconfig.yml +# name you'd use to refer to this instrument +sed -i s+"instrument_name: BCDA EPICS Bluesky training"+"instrument_name: APS 44ID XYZ"+g instrument/iconfig.yml +# set a default value for proposal ID (update from shell for each new proposal's beam time) +sed -i s+"proposal_id: training"+"proposal_id: commissioning"+g instrument/iconfig.yml ``` -cd "${BLUESKY_DIR}/profile_bluesky" -wget https://github.com/BCDA-APS/use_bluesky/raw/main/install/install_startup.sh + +#### Remove training repository components + +Rather than deleting the content, it is commented out +so the configuration and files described may be used +as reference. + +The training repo uses some specific IOCs. Comment that out. + +```bash +sed -i s+"^AD"+"# AD"+g instrument/iconfig.yml +sed -i s+"^GP"+"# GP"+g instrument/iconfig.yml +sed -i s+"^BLUESKY_MOUNT_PATH"+"# BLUESKY_MOUNT_PATH"+g instrument/iconfig.yml ``` -Run the installer +The training repo uses some specific devices from those IOCs. Comment that out. +```bash +sed -i s+"^from ."+"# from ."+g instrument/devices/__init__.py ``` -bash ./install_startup.sh BEAMLINE INSTRUMENT REPOSITORY + +Comment out any example plans. + +```bash +sed -i s+"^from ."+"# from ."+g instrument/plans/__init__.py ``` -where `BEAMLINE` `INSTRUMENT` `REPOSITORY` each contain no white space -characters. The terms BEAMLINE and INSTRUMENT will be added to the standard -metadata added to every Bluesky -[_run_](https://blueskyproject.io/bluesky/documents.html?highlight=run). -Example: `bash ./install_startup.sh 45-ID FemtoScanner 45id_femtoscanner` +### Install -At the APS, the beam line controls group (BCDA) will assign the -REPOSITORY and its SERVER. +```bash +cd /tmp/bluesky_training +tar cf - bluesky | (cd "${HOME}" && tar xf -) +cd /tmp +/bin/rm -rf /tmp/bluesky_training +cd "${HOME}" +mkdir -p bin + +cd "${HOME}"/bin +ln -s "${HOME}"/bluesky/blueskyStarter.sh ./ + +cp \ + "${HOME}"/bluesky/console/__start_bluesky_instrument__.py \ + "${HOME}"/.ipython-bluesky/profile_bluesky/startup/ +``` ## Commit Instrument Package to Version Control @@ -397,7 +501,7 @@ The [`apstools`]() package has an application that will translate most of the SPEC config file into ophyd commands. The output is to the console. Use a pipe to direct the output to a new file: -``` +```bash export INSTRUMENT_DIR=${BLUESKY_DIR}/profile_bluesky/startup/instrument spec2ophyd CONFIG_FILE | tee ${INSTRUMENT_DIR}/devices/spec.py ``` @@ -409,8 +513,8 @@ imports there.
spec2ophyd example: -``` -(bluesky_2021_1) mintadmin@mint-vm:/tmp$ spec2ophyd /home/mintadmin/Documents/projects/BCDA-APS/apstools/apstools/migration/config +```bash +(bluesky_2022_3) mintadmin@mint-vm:/tmp$ spec2ophyd /home/mintadmin/Documents/projects/BCDA-APS/apstools/apstools/migration/config """ ophyd commands from SPEC config file @@ -501,18 +605,19 @@ I000 = c0.channels.chan06.s ## Add Environment Configuration to .bash_aliases -The `~/.bash_aliases` file is the usual place to customize the bash shell. It -is executed from `~/.bashrc` when a new console is created. +The `~/.bash_aliases` file is the usual place to customize the bash shell. If +it exists, it is executed from `~/.bashrc` when a new console is created. -Add these lines to `~/.bash_aliases`: +Add these lines to `~/.bash_aliases` (or `~/.bashrc` if the other does not exist): ``` -export CONDA_ENVIRONMENT=bluesky_2021_1 +export CONDA_ENVIRONMENT=bluesky_2022_3 alias become_bluesky='conda activate "${CONDA_ENVIRONMENT}" ' ``` ## Install Starter Script + The commands and settings to configure the local environment to begin a Bluesky session can be arranged by a simple shell script that starts the session. A sample starter has been provided: @@ -536,12 +641,7 @@ and create the directory: mkdir ~/bin ``` -Add the starter script to `~/bin`: - -``` -cd ~/bin -ln -s ${BLUESKY_DIR}/profile_bluesky/startup/blueskyStarter.sh ./ -``` +The starter script was added above, in the [Install](#install) section. TIP: You might rename `~/bin/blueskyStarter.sh` to something appropriate for your instrument name, such as the hypothetical example above: @@ -551,3 +651,7 @@ your instrument name, such as the hypothetical example above: * Change to desired working directory. * Start Bluesky session using the starter script. + +```bash +blueskyStarter.sh +``` diff --git a/install/install_startup.sh b/install/install_startup.sh deleted file mode 100755 index d0fd1bb..0000000 --- a/install/install_startup.sh +++ /dev/null @@ -1,97 +0,0 @@ -#!/bin/bash - -# file: install_startup.sh -# purpose: -# install Bluesky IPython startup directory -# with default instrument package into `pwd` -# (within a IPython profile directory). -# -# EXAMPLE -# cd ~/.ipython-bluesky/profile_bluesky -# path/to/install_startup.sh 45ID WNI wnicat -# -# (base) jemian@wow ~/.ipython/profile_testing $ /home/beams/JEMIAN/Documents/projects/BCDA-APS/use_bluesky/install/install_startup.sh 45ID WNI wnicat -# Extracting instrument package template: '/home/beams1/JEMIAN/Documents/projects/BCDA-APS/use_bluesky/install/2021_1_startup.tar.gz' -# Installing to IPython profile directory: '/home/beams1/JEMIAN/.ipython/profile_testing/profile_testing' -# setting beam line name: 45ID -# setting instrument name: WNI -# setting databroker catalog: wnicat -# editing starter shell script: /home/beams1/JEMIAN/.ipython/profile_testing/startup/blueskyStarter.sh -# IPython directory: /home/beams1/JEMIAN/.ipython -# IPython profile: testing - - -# assumes tarball is in same directory as this bash script -export TARNAME=2021_1_startup.tar.gz -export STARTUP_TARBALL="$(realpath $(dirname $0))/${TARNAME}" -# or get from web site -export REPO=https://github.com/BCDA-APS/use_bluesky -export STARTUP_URL="${REPO}/raw/main/install/${TARNAME}" - -# pip hangs after running any command -# https://github.com/pypa/pip/issues/9061 -export PYTHON_KEYRING_BACKEND=keyring.backends.null.Keyring - - -function show_usage() { - echo "usage: $0 BEAMLINE INSTRUMENT CATALOG" -} - -function perform_installation_tasks() { - path=$(basename $(realpath $(pwd))) - if [[ "$(echo $path | cut -f 1 -d '_')" != "profile" ]]; then - echo "Not an IPython profile directory: '${path}'" - echo "No installation." - return - fi - # Will only proceed if in IPython configuration directory - # Name style: profile_ABCDEFG - PROFILE=$(echo $path | cut -f 2 -d '_') - IPATH=$(dirname $(realpath $(dirname ${path}))) - - if [ -d "startup" ]; then - echo "Existing 'startup'. Will not modify." - return - # /bin/rm -rf ./startup - fi - - if [ -f "${STARTUP_TARBALL}" ]; then - echo "Extracting instrument package template: '${STARTUP_TARBALL}'" - tar xzf ${STARTUP_TARBALL} - else - # If can't get the tarball locally, then get from URL - echo "Extracting instrument package template: '${STARTUP_URL}'" - wget ${STARTUP_URL} ./ - tar xzf ${TARNAME} - rm ${TARNAME} - fi - - echo "Installing to IPython profile directory: '$(realpath ${path})'" - BEAMLINE=$1 - INSTRUMENT=$2 - CATALOG=$3 - - echo "setting beam line name: ${BEAMLINE}" - sed -i s+"BEAMLINE_NAME"+"${BEAMLINE}"+g startup/README.md - sed -i s+"BEAMLINE_NAME"+"${BEAMLINE}"+g startup/instrument/framework/metadata.py - - echo "setting instrument name: ${INSTRUMENT}" - sed -i s+"INSTRUMENT_NAME"+"${INSTRUMENT}"+g startup/README.md - sed -i s+"INSTRUMENT_NAME"+"${INSTRUMENT}"+g startup/instrument/framework/metadata.py - - echo "setting databroker catalog: ${CATALOG}" - sed -i s+"CATALOG_NAME"+"${CATALOG}"+g startup/instrument/framework/initialize.py - - # edit starter script - echo "editing starter shell script: $(realpath startup/blueskyStarter.sh)" - echo "IPython directory: ${IPATH}" - sed -i s+"IPYTHONDIR=~/.ipython-bluesky"+"IPYTHONDIR=${IPATH}"+g startup/blueskyStarter.sh - echo "IPython profile: ${PROFILE}" - sed -i s+"IPYTHON_PROFILE=bluesky"+"IPYTHON_PROFILE=${PROFILE}"+g startup/blueskyStarter.sh -} - -if (( $# == 3 )); then - perform_installation_tasks $1 $2 $3 -else - show_usage -fi diff --git a/instrument_package_guide.md b/instrument_package_guide.md index a6c8a27..b857281 100644 --- a/instrument_package_guide.md +++ b/instrument_package_guide.md @@ -12,6 +12,7 @@ reference for ideas rather than a set of requirements. **Contents** - [Instrument Package Guide](#instrument-package-guide) - [Basic Structure](#basic-structure) + - [Instrument Configuration File](#instrument-configuration-file) - [Add Motor(s)](#add-motors) - [Add Scaler(s)](#add-scalers) - [Re-organize into Devices](#re-organize-into-devices) @@ -29,6 +30,7 @@ these directories may contain python file(s). ``` instrument/ <-- all aspects of the equipment + iconfig.yml <-- configuration details specific to your instrument callbacks/ <-- your custom Python code that responds to RunEngine documents or other devices/ <-- your equipment and its connections with EPICS framework/ <-- Configure the Bluesky framework @@ -48,6 +50,42 @@ that identifies which Python symbols from that file will be imported by default. (Any Python symbol may be imported by name, this just controls the more general case.) +## Instrument Configuration File + +To make it easier for teams to manage the instrument configuration, certain +details of the instrument configuration (such as timeout durations and default +metadata) are described in the file `instrument/iconfig.yml`. This is a text +file in [YAML](https://yaml.org) format. Some of these details are used when +connecting to the databroker, others are used when configuring device or plan +details. + +For example, these keys: + +```yml +ADSIM_IOC_PREFIX: "ad:" +AD_IMAGE_DIR: "adsimdet/%Y/%m/%d" +AD_MOUNT_PATH: /tmp +BLUESKY_MOUNT_PATH: /tmp/docker_ioc/iocad/tmp +``` + +are used in the area detector setup (of the training example): + +```py +from .. import iconfig +# ... +IOC = iconfig["ADSIM_IOC_PREFIX"] +IMAGE_DIR = iconfig["AD_IMAGE_DIR"] +AD_IOC_MOUNT_PATH = pathlib.Path(iconfig["AD_MOUNT_PATH"]) +BLUESKY_MOUNT_PATH = pathlib.Path(iconfig["BLUESKY_MOUNT_PATH"]) +# ... +adsimdet = MySimDetector(IOC, name="adsimdet", labels=("area_detector",)) +``` + +Feel free to add keys (following the rules for YAML files). +You might refer to another +[example `iconfig.yml` file](https://github.com/BCDA-APS/bdp_controls/blob/main/qserver/instrument/iconfig.yml) +for additionaL ideas. + ## Add Motor(s) To add an EPICS [motor](https://github.com/epics-modules/motor) to your @@ -76,7 +114,7 @@ CAUTION: Avoid certain names such as [`del`](https://docs.python.org/3/reference/simple_stmts.html?highlight=del#the-del-statement) and [`lambda`](https://docs.python.org/3/reference/expressions.html?highlight=lambda#lambda) -since these are reserved Python names with very important meanings. +since these are reserved Python names with very important meanings. Be sure to add this new motor symbol to the `__all__` list near the top of the file. @@ -294,8 +332,8 @@ print(f"m1 is now at {m1.position:.3f}") Start the bluesky session: ``` -(base) prjemian@zap:/tmp/demo$ blueskyZap.sh -Python 3.8.5 (default, Sep 4 2020, 07:30:14) +(base) prjemian@zap:/tmp/demo$ blueskyZap.sh +Python 3.8.5 (default, Sep 4 2020, 07:30:14) Type 'copyright', 'credits' or 'license' for more information IPython 7.20.0 -- An enhanced Interactive Python. Type '?' for help. @@ -325,7 +363,7 @@ I Sun-15:20:23 - to change SPEC file, use command: newSpecFile('title') I Sun-15:20:23 - /home/prjemian/.ipython/profile_bluesky/startup/instrument/devices/motors.py I Sun-15:20:23 - /home/prjemian/.ipython/profile_bluesky/startup/instrument/devices/scaler.py -In [1]: +In [1]: ``` Run the `my_code.py` file: @@ -334,7 +372,7 @@ Run the `my_code.py` file: In [1]: %run -i my_code.py m1 is now at 3.000 -In [2]: +In [2]: ```