-
Notifications
You must be signed in to change notification settings - Fork 11
/
Copy pathdiscovery_3-storage.qmd
146 lines (79 loc) · 7.19 KB
/
discovery_3-storage.qmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
# Lotterhos Lab vs. Personal Storage on Discovery and GitHub
[Edit this page](https://github.com/DrK-Lo/lotterhoslabprotocols/edit/main/discovery_3-storage.qmd)
------------------------------------------------------------------------
- We have **active** storage (10 TB) that should be used for 'active' projects, and **archival** storage for storing raw files and outputs from projects. Please be smart about how you use archival storage and work with Dr. Lotterhos to make sure only relevant files are being archived.
- In addition each user has a home directory and a scratch directory.
## Storage unix tricks
`df -h /work/lotterhos` #total space useage for lotterhos
`pwd | du -h --max-depth=1` get disk use for folders in the current directory
`ls -lah` view owners of files and file sizes (the total size of the folders is not displayed)
## **Working with github:** {.unnumbered}
In Github Settings, go down to "Developer settings" and create a personal access token:
Make sure it is "Classic" and then "configure SSO".
More information here:
[Authorizing personal access token](https://docs.github.com/en/enterprise-cloud@latest/authentication/authenticating-with-saml-single-sign-on/authorizing-a-personal-access-token-for-use-with-saml-single-sign-on)
Then in the command line in Discovery use your token when it asks for a password.
It is good practice to create a repo on Github and then clone it to Discovery to start a project. You can follow instructions in Buffalo's Bioinformatics book to work with git on linux.
In addition, you may want to collaborate on Discovery through your git repo.
You may also need to set up ssh keys:
example: When person A started working on his git branch in Discovery, he cloned that branch to Discovery (e.g in the folder and repo /work/lotterhos/MVP-NonClinalAF). To give person B access,
1. Person-B Go to the terminal and copy your public key file content.
cat ~/.ssh/id_rsa.pub
(If you don't have a key pair generated already, generate a new one using command "ssh-keygen" )
2. Go to the github website and login with Person-B credentials.
3. In the upper-right corner of the github page, click Person-B profile photo, then click Settings.
4. Click New SSH key or Add SSH key.
5. Paste your key into the "Key" field.
6. Click Add SSH key.
7. Test the connection by running the following command in the terminal on person B Discovery:
ssh -T git@github.com
### Giving someone access to a folder and all containing folders
General code:
`setfacl -R -m u:username:rwx myfolder`
Example for user `rumberger.c` in folder `MVP_Genotypes_DataMerge`:
`setfacl -R -m u:rumberger.c:rwx MVP_Genotypes_DataMerge`
### **Your home directory on Discovery\*\*.** {.unnumbered}
- Path: `/~/` or `/home/username`
- Note that only you have access to these files.
### **Your scratch directory on Discovery\*\*. Use this for intermediate outputs that are not backed up.** {.unnumbered}
- Path: `/scratch/username`
- Note that only you have access to these files.
### **Lotterhos shared storage on Discovery** {.unnumbered}
- /work/lotterhos/
- Katie will create a folder for each lab project in this directory. For each folder, Katie will submit a ticket to research computing to ask them to create a group for the project with members who will have access to that folder.
Example:
mkdir BlackSeaBass # make the folder
chgrp -R lotterhos-bsb 2020_NOAA_BlackSeaBass_ddRADb # change the group on the directory. I forget what the -R does
chmod -R 775 2020_NOAA_BlackSeaBass_ddRADb # give group permissions to the directory
ls -la 2020_NOAA_BlackSeaBass_ddRADb # check permissions
- Groups
- `lotterhos` - all Lotterhos Lab members
- `lotterhos-bsb` - access to Black Sea Bass project
- `lotterhos-cod`
- Groups are created by research computing
Find the groups of a user: `groups tbittar` will give the groups for user tbittar
Find the users in a group: `users lotterhos-bsb lotterhos` will give the users for group `lotterhos-bsb` and group `lotterhos`
`usermod -a -G groupname username` I forget what this does
Sometimes I have still had trouble making the code work on existing directories, but it can work if you copy the directory to a new one
### **Archival Lotterhos Lab storage\*\* (25 TB for lab). Northeastern NESE collection.** {.unnumbered}
- Path: `/lotterhosarchive/`
- Access through GLOBUS endpoint that Katie can Share with you
# Outdated storage
### **Active Lotterhos Lab storage: more information\*\* (? TB for lab).** {.unnumbered}
- smb://nunet.neu.edu/rc-shares/lotterhos/
- Ask Katie to share a Globus endpoint with you. The endpoint with include the words "smb Volumes/lotterhos"
- Within this storage space we have a shared folder for each project, that only the people on that project have access to.
- For this endpoint to work, it has to be mapped through Katie's laptop or the iMac in the lab. *Katie's laptop from home: sign into NU VPN, make sure Globus Connect Personal App is running, plug in ethernet cord*
- Because accessing this server is done through a computer, it makes transferring files specific to the computer that the server is mapped to. Typically, you will ask Katie to share the Globus endpoint for this server that is set up through the lab iMac. As long as the server is mapped to the computer (typically lab iMac) and that connection is not interrupted, you should have no problems.
- From a laptop or over wifi, transfers of large files are very slow. This is why we map to the server through the iMac. The point of all the steps we outline below is so we can use Globus to transfer files from the smb drive to Discovery.
- To connect to the smb server, follow these directions to map to the server on a computer this document
- Setting up a Globus endpoint for the folder to share (typically Katie only)
- *Intitial step if Globus personal endpoint for the computer is not set up*: Download and install the Globus Connect Personal App. In Globus web app, click on "Add a personal endpoint" and enter the key in the Desktop app.
- *Intial setup if working from a home laptop*: Use GlobalProtect to establish a VPN connection (vpn.northeastern.edu) (if you are having trouble establishing the VPN connection on a Mac, check your Security and Permissions settings and give the app permissions, see image). Then you should be able to map to the server.
- Step 1: In the Globus Desktop App, then go to Preferences-\> Access and click the "+" sign. On a Mac, at the top center click on the bar, click on the name of the computer, and you should see the mapped server there as a volume `Volumes/lotterhos`. Click on it and add it to the list.
- Step 2: In the Globus Web App, go to the endpoint for the computer. Then type `Volumes/lotterhos/` in the path. Save it as a bookmark.
- Step 3: Before you can conduct file transfers from a personal endpoint, you need to have a "Globus Plus Account". Email James Macalino at NU research computing to get added. You will then receive an email.