Skip to content

Commit

Permalink
minor updates
Browse files Browse the repository at this point in the history
  • Loading branch information
kurtshowmaker committed Mar 11, 2024
1 parent 74525d5 commit 4029d4a
Show file tree
Hide file tree
Showing 17 changed files with 10,615 additions and 121 deletions.
25 changes: 18 additions & 7 deletions config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,32 +8,32 @@
# lc: Library Carpentry
# cp: Carpentries (to use for instructor training for instance)
# incubator: The Carpentries Incubator
carpentry: 'incubator'
carpentry: 'none'

# Overall title for pages.
title: 'Lesson Title' # FIXME
title: 'Singularity Workshop 2023' # FIXME

# Date the lesson was created (YYYY-MM-DD, this is empty by default)
created: ~ # FIXME

# Comma-separated list of keywords for the lesson
keywords: 'software, data, lesson, The Carpentries' # FIXME
keywords: 'software, data, lesson' # FIXME

# Life cycle stage of the lesson
# possible values: pre-alpha, alpha, beta, stable
life_cycle: 'pre-alpha' # FIXME
life_cycle: 'none' # FIXME

# License of the lesson
license: 'CC-BY 4.0'

# Link to the source repository for this lesson
source: 'https://github.com/carpentries/workbench-template-md' # FIXME
source: 'https://github.com/TheJacksonLaboratory/singularity-workshop-2023' # FIXME

# Default branch of your lesson
branch: 'main'

# Who to contact if there are any issues
contact: 'team@carpentries.org' # FIXME
contact: 'kurt.showmaker@jax.org' # FIXME

# Navigation ------------------------------------------------
#
Expand All @@ -59,7 +59,18 @@ contact: 'team@carpentries.org' # FIXME

# Order of episodes in your lesson
episodes:
- introduction.md
- 00-introduction.md
- 01-intro-to-computers.md
- 02-singularity-commands.md
- 03-singularity-pull-hello-world.md
- 04-repos-and-registries.md
- 05-singularity-build.md
- 06-bioinformatics-qc.md
- 07-bioinformatics-bwa.md
- 08-circos.md
- 09-singularity-build-blast-example.md
- 10-singularity_build_rstudio.md


# Information for Learners
learners:
Expand Down
36 changes: 36 additions & 0 deletions episodes/00-introduction.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
---
title: "00-introduction"
teaching: 10
exercises: 2
---

## Introduction

This workshop is designed for a beginner with little to now

By the end of this workshop you will have:

- Created 16 contianers.

- Executed 12+ commands with containers.

- Build 2 custom containers.

- Build 2 websites and viewed them with containers.

- Run the most common singularity commands (pull, build, exec)

- Run the exec command command with bind mounts to access data.

- Encounter 3 gotchas and discuss solutions to deal with them.

- Gotcha 1: Repositories are managed by other users.
- Solution 1: Keep good documentiaton on how your software was built.

- Gotcha 2: Cant mount the current working directory.
- Solution 2: Try the **-B $PWD** flag.

- Gotcha 3: Cant mount the the directory with the data.
- Solution 3: Try the **-B <my_data_dir_path>:<my_data_dir_path>** flag.


55 changes: 55 additions & 0 deletions episodes/01-intro-to-computers.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
---
title: "01-intro-to-computers"
teaching: 30
exercises: 0
---

![parts example](episodes/fig/01_computer_parts.svg){alt='A collection of computer parts including disks, ram, cpu'}

## Computer Component Review

- **CPU:** CPUs are the data process unit, they are composed of multiple cores. For legacy reasons software often refers the number of cores as the number of CPUs, so yeah that is confusing.

- **RAM (a.k.a MEMORY):** RAM is fast digital storage. Most programs utilize RAM for access to data needed more than once. RAM is generally non-persistent when the powered off RAM memory is lost.

- **DISK:** Disk is persistent digital storage that is not as fast as RAM. Disk storage can be made up of one or more disks such as hard drives (HDD) and/or Solid State Harddrives (SSD). Multiple disk can be configured together for increased performance and drive failure protection.

- **NETWORKING:** Switches and network access cards within computers allow for computers to be networked together.

- **GPU:** A Graphics Processing Unit (GPU) is a computer component that is capable of rendering graphics, these are also useful for conducting certain mathematical calculations.

## Consumer Computer vs Servers vs HPC vs Sumhpc

| Component | Home/Busines Computer | Server | Typical Individual Node in HPC | Typical Total HPC System | Individual Node on Sumhpc | Total Sumhpc System |
|-----------|-----------------------|------------|-------------------------------|--------------------------|---------------------------|---------------------|
| CPU (cores)| 4 - 8 | 12 - 128 | 32 - 128 | 1000s | 70\* | 7,000 |
| RAM(GB) | 8 -16 | 64 - 960 | 240 - 3000 | 64,000 | 754 - 3TB | 76.8 TB|
| DISK (TB)| .5 - 1 TB | 8 - 100 | None - 1 TB | 100s (Networked) | NA | 2.7 PB |
| Networking (Gbe)| .1 - 1 | 1 - 10 | 40 - 100 | 40 - 100 | 40 | 40 + |


## Computer Ports

A port is a communication endpoint.

## Introduction to OS, Virtual Machines and Containers


Why can't I use Docker?
Docker images are not secure because they allow users to gain root access to the compute nodes. Singularity effectively runs as the user running the command and does not result in elevated access. Also, docker interacts with the slurm job scheduler in a way that causes resource requests and usages to not match up, making it difficult to keep job queueing fair for all users. In that the clusters are multi-user systems, we want to make sure people can work without worry that others are accessing their data or unfairly using up resources.

## Important notes on how they relate to singularity

-CPU There are 2 common CPU architectures in modern systems x86_64 and ARM. Singularity containers are architecture specific.
```--arch string architecture to pull from library (default "amd64")```

### citations

Lesson adapted from:

https://github.com/TheJacksonLaboratory/intro-to-hpc


https://crc.pitt.edu/singularity


39 changes: 39 additions & 0 deletions episodes/02-singularity-commands.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
---
title: "02-singularity-commands"
teaching: 30
exercises: 0
---


### Fequently used singularity commands

- **singularity pull**

- **singularity shell**

- **singularity cache**

- **singularity build**

- **singularity exec**

**These hands on tutorial are being conducted on a base Centos7 OS VM with the following programs installed.**

- apptainer (including the alias singularity)

- nano

- wget

- unzip

**The following ports have been made available to the system:**

- 8080,8787,8789

**The following folders have been setup on the remote system:**

- /projects/my-lab
- /flashscratch
- /workshop_data

Loading

0 comments on commit 4029d4a

Please sign in to comment.