$
indicates a command is run in the shell - you don't need to write this!
Shell uses space to separate things - this can complicate other times we want to use spaces
- expands by default (scripts will break with spaces)
- quotes to control expansion
$ "this is fine"
$ this will cause problems
Which directory am I in?
$ pwd
What is in this directory?
$ ls
Useful options for ls
:
-a show hidden files
-G colorized
-l long format
-r reverse order
-t sort by time
Tree is a very useful program for showing the folder structure deeper in the file system
$ tree -L {num}
Clear the terminal
$ clear
Make an empty file
$ touch myfile.txt
Edit a file
$ nano myfile.txt
# print file contents
$ cat readme.md
# print first n rows
$ head -n 3 readme.md
# print last n rows
$ tail -n 3 readme.md
# paging over the file
$ less readme.md
Make a directory
$ mkdir practice-dir
We can also recursively create directories
$ mkdir -p practice/subfolder
Be careful with mv
- it will overwrite the file!
$ cp myfile.txt practice-dir/copy.txt
$ mv myfile.txt practice-dir/my-file.txt
Move down into a folder
$ cd practice-dir
To go one level up
$ cd ..
Go back to last directory
$ cd -
Getting to highest level folders by cd /etc
, cd /Users
Going to the home folder (/Users/adam
)
$ cd ~
$ cd $HOME
$ cd
Use the last argument from your previous command
$ mkdir mydir
$ tree $_
Run your last command
$ !!
Show sizes of files in directory (the *
symbol is a wildcard that matches everything)
$ du -hs *
Show disk usage
$ df -h
Make a file with a single line tdd
$ echo tdd > .python-version
Be careful with rm
- there is no trash can for rm
!
$ rm file
$ rm -rf directory
You can use the default program in your OS to open files
# open the current folder in your OS's file explorer
$ open .
TAB
for autocomplete
Up and down arrows to move between commands
$ !python
- rerun last command for program
$ history
will print the history of your shell (the file is located at ~/.bash_history
)
$ <C-r>
to search history
Aliases are also very important for improving speed (see below)
The terminal is a stateful system - it has a whole bunch of variables. You can see these using env
(the equivalent of dir()
in Python)
We can set a variable using:
$ zz=1
If I want to see what this variable is (note the $
- this is always used)
$ echo $zz
But these variables will not be inherited by sub processes - to get this we need to use export
. You will always see export
used in the shell config scripts (i.e. .bash_rc
), and the simpler variable assignment in shell scripts.
$ export zz=2
$ man ls
$ which ls
Locations the shell checks when you type a command. You can think of adding a location to your PATH
as installing a program.
$ echo $PATH | tr ":" "\n"
$ export PATH=$PATH:$SPARK_HOME/bin
A common pattern you will see in install scripts
$ echo 'export PATH=$PATH:$SPARK_HOME/bin' >> ~/.bashrc
Telling the system which program to use to run the file
#!/usr/bin/env bash
Often will need to change permissions
$ chmod +x myfile
Now can run file without specifying the program
$ ./myfile
Running file / loading a file into your bash environment
$ source myfile
Your shell is configured using
~/.bashrc
(Linux)
~/.bash_profile
(Mac - but Mac will also have a bashrc
)
These scripts are run every time you start a new shell. It is common to add echo
statements to these files to get used to this idea.
RC = a run command
- file that contains startup infomation
"ls" to run without alias expansion
Useful aliases
alias ls='ls -aGl'
alias exut='exit'
alias eixt='exit'
alias exot='exit'
alias ext='exit'
alias eit='exit'
alias q='exit'
alias c='clear'
alias cls='clear && ls'
alias ctree='clear && tree'
alias bashrc='vim ~/git/dotfiles/.bashrc'
alias gs='git status'
alias ga='git add -u'
alias gc='git commit -m '
alias gp='git push origin '
alias gls='clear && git status'
Generate project structure for use in a readme
# append
$ tree >> out.txt
# new file
$ tree > out.txt
Another common pattern is generating a requirements file for a Python project
$ pip freeze > requirements.txt
You can also put the redirection at the start
$ > out.txt tree
grep
is one of the UNIX searching tools.
Search for the string "MyClass" inside a file, recursively, and return a list of files:
$ grep -rl "MyClass"" .
You can search by filename:
$ find . -name ".bashrc"
$ pip freeze | grep numpy
$ grep -rl LSTM . | grep -v __pycache__ | grep -v .ipynb_checkpoints
Very important skill to work on remote machines
The basic syntax for SSH is
$ ssh USER@HOSTNAME
Commonly if you are using AWS you will SSH while also passing your private key
$ ssh -i ~/.ssh/key.pem -tt ubuntu@ec2-54-93-188-105.eu-central-1.compute.amazonaws.com
$ ssh -N -L localhost:8888:localhost:8888 $USER@$HOST
Common to backup your dotfiles on GitHub - you can find my dotfiles here.
- you can clone this onto a remote machine & source your dotfiles!
Downloads files served with HTTP, HTTPS, or FTP over a network
$ wget https://news.ycombinator.com/y18.gif
$ cat raw-data/specs.csv | column -t -s, | less -S