set up example for Relion on AWS ParallelCluster
- Validated conditions
- AWS ParallelCluster 3.1.3
- Relion 3.1.3
us-east-1
region
⚠️ This tutorial environment does not include FSx for Lustre / S3 integration
On AWS CloudShell, run following commands for cluster creation. The create_relion_cluster.sh
script use us-east-1
region in default.
You could change AWS_REGION
parameter in the script.
git clone https://github.com/DaisukeMiyamoto/aws-parallelcluster-relion
cd aws-parallelcluster-relion/01_setup_cluster
./create_relion_cluster.sh
source ~/.bashrc
You could check cluster status with following command.
Cluster creation usually needs 5 to 10 minutes.
If the clusterStatus
become CREATE_COMPLETE
from CREATE_IN_PROGRESS
, go to next step.
pcluster list-clusters
Run following command and open published URL to access NICE-DCV remote desktop environment.
pcluster dcv-connect --cluster-name ${PCLUSTER_CLUSTER_NAME} --key-path ~/.ssh/${SSH_KEY}
- If it is your first time to access the cluster, a message like
The authenticity of host '123.123.123.123 (123.123.123.123)' can't be established.
could be shown. You need to typeyes
and hitting enter. - On opening the NICE-DCV URL with browser, you will face
Your connection is not private
warning e.g. image. You need to clickAdvanced
andProceed to <IP> (unsafe)
to open NICE-DCV desktop.
On NICE-DCV desktop, open terminal from Activities button on top-left corner. Run following commands on the terminal for installing Relion.
cd /shared
git clone https://github.com/DaisukeMiyamoto/aws-parallelcluster-relion
cd aws-parallelcluster-relion/02_relion_gui
chmod +x ./setup_relion_v31_gui.sh
./setup_relion_v31_gui.sh
source ~/.bashrc
⚠️ This compilation settings are not optimized. You need to customize for appropriate benchmarkings.
Run following commands to download datasets and launch Relion. This dataset include all completed steps and intermediate files. You could run any process without input parameter settings. Compilation needs few minutes.
⚠️ This step downloads large amount of data (7GB) from Osaka University. Please read guides for appropriate use.
cd /shared
wget http://www.protein.osaka-u.ac.jp/rcsfp/databases/members/kawabata/EMtutorial/EMPIAR-10248/EMPIAR-10248_tutorial_precalculated_results.tar.gz
tar xvf EMPIAR-10248_tutorial_precalculated_results.tar.gz
cd EMPIAR-10248_tutorial
relion &
You could submit a job to job scheduler with Running
tab settings.
This tutorial shows an example about running Class2D process with batch job way.
- Click
007:Class2D/ManuManuPick/
inFinished jobs
window to re-load previous settings. - Click
2D classification
in top-left corner to create a new settings. - Open
Compute
tab and set following parameters:- Use GPU acceleration?: No
- Open
Running
tab and set following parameters:
- [Running] Tab
- Number of MPI procs: 1
- Number of threads: 16
- Submit to queue?: Yes
- Queue name: c6i-4xlarge
- Queue submit command: sbatch
- Standard submission script:
/shared/aws-parallelcluster-relion/02_relion_gui/gui_batch_template_slurm.sh
- Click
Run!
button to submit a new job.- After job submission, you could find new EC2 instance is launching on EC2 management console.
- You could also find your job status with
squeue
command. Usually, launching instance needs almost 4 minutes. - When the job started,
ST
(STATUS) becomeR
(RUNNING) fromCF
insqueue
command. You could also check the job progress with output box on Relion ( image ).
- This job takes arround 10 minutes.
- This figure show how relion job submission works with GUI and template files.
Relion GUI replace parameters (e.g.
XXXmpinodesXXX
) with GUI settings.
TBU
cd /shared/aws-parallelcluster-relion/03_relion_benchmarks
chmod +x 03_download_benchmarks.sh
./03_download_benchmarks.sh
chmod +x 01_setup_dependencies.sh
./01_setup_dependencies.sh
TBU
On the terminal on CloudRun, run following command to delete cluster. This step delete all data in the cluster including FSx for Lustre.
pcluster delete-cluster -n ${PCLUSTER_CLUSTER_NAME}