Meet Your Local Terminal¶
Lecture Notes¶
⬇︎ Terminal
⬇︎ Linux Cheat-sheet
If you are new to the command line you might find these links useful:
- The Linux Command Line (Book and Website)
- How to use the Command Line | Terminal Basics for Beginners (Movie)
- Linux Tutorial
Get Started¶
Open your terminal and get started. The gray boxes are code chunks and you can copy past the content. Click on the page icon (⎘) at the top-right corner of each code block to copy the content directly into memory. Every command after the octothorp # (hashtag, pound or number sign, sharp symbol) is a comment and will be ignored. The number of octothorp does not matter and can be used to structure your code.
Syntax¶
The general pattern of a command line interface is:
prompt> command [option(s)] (File)
Example: List (ls) content of a folder.
ls # content of current folder ls test/ # content of folder test ls -lh # list content of current folder use human-readable file size
The list command ls is providing more than just the directory content. It shows you the rights, owner, size, and (last changing) date.
# - rw- r-- r-- jwalser users 161K Jan 17 12:22 p117_Help.txt # - rw- r-- r-- jwalser users 92M Jan 17 16:51 p117_Metafile.txt # d rwx r-x r-x jwalser users 16G Feb 19 15:43 data # | --- --- --- | | | | | # | | | | | | | | └⊳ files / folders # | | | | | | | └⊳ date and time # | | | | | | └⊳ size # | | | | | └⊳ group # | | | | └⊳ owner # | | | └⊳ rights all (read/write/execute) # | | └⊳ rights group (read/write/execute) # | └⊳ rights owner (read/write/execute) # └⊳ d director / - file
Have a look at the Guru99 tutorial for more details about file permissions.
Basic Commands¶
pwd.................: absolute pathname of the current working direction man <command>.......: manual page for command (exit with q) cd <where>..........: change directory/folder cd .. ..............: go up one directory cd..................: go home mkdir <dir>.........: create directory rmdir <dir>.........: remove directory (if empty) ls <dir>............: list content of directory ls -alh <dir>.......: more detail list echo "message"......: prints content or message cat <file>..........: print and concatenate files head -n 5 <file>....: show first n lines tail -n 5 <file>....: show last n lines more <file>.........: read file (exit with q) less <file>.........: similar to more but newer > & >>..............: re-direct output (e.g. pwd > file.txt) cp <ori> <copy>.....: copy file mv <old> <new>......: move and/or rename file rm <file>...........: remove file - careful!!! wc <file>...........: word, line, character, and byte count grep "query" <file>.: search file(s) matching query find................: find files based on characteristics (e.g. name) sed.................: transform content of text files clear...............: clear terminal history.............: show terminal history date................: display date cal.................: display calendar
▻ Have a look at the Linux cheat-sheet for more commands and ideas.
Help¶
No worries if you do not know the command, help is at your fingertip! The manual can help you to understand the command better. You could Google it but you might get information not suitable for your system/version.
## Help (Syntax: help <command>) help cat # might not work for all commands ## Help cat --help # might not work for all commands ## Access the manual (Syntax: man <command>) man cat # example ## Version cat --version # might not work for all commands ## Where is binary file for the command located? which cat
BUILT-IN Variables¶
echo ${SHELL} # Gives present shell echo ${USER} # Displays username echo ${HOME} # Home directory of User echo ${RANDOM} # To get a random number echo ${PWD} # Current directory
User Defined Variables¶
MyRegistry="NCC-1701" MyName="USS_Enterprise" echo "${MyName} (${MyRegistry})"
Create a Working Directory¶
## Where are you? Get current location: pwd # structure: /<folder>/<folder>/<folder> ## Create a working directory mkdir ${HOME}/GDA20 # Main folder (directory) in your home directory mkdir ${HOME}/GDA20/Terminal # Sub-folder (sub-directory) inside you main folder mkdir ${HOME}/GDA20/Terminal # Create yet another folder inside this folder # HOME < you are here # └── GDC20 # └──Terminal ## "Go to" the working diretory cd ${HOME}/GDA20/Terminal # Change directory # HOME # └── GDC20 # └──Terminal < now you are here
Navigation¶
Most of the time, there a multiple solutions for a specific task. For example, before we used mkdir multiple time to create subfolders. Would it not be nicer to do this with less typing?
## Test Folders mkdir -p TestFolder_A/TestFolder_B/TestFolder_C # ➜ Option -p creates intermediate folders # WD < you are here # └── TestFolder_A # └── TestFolder_B # └── TestFolder_C ## Go down and go up / step-by-step cd TestFolder_A/TestFolder_B/TestFolder_C # WD # └── TestFolder_A # └── TestFolder_B # └── TestFolder_C < you are here cd .. # go one folder up # WD # └── TestFolder_A # └── TestFolder_B < you are here # └── TestFolder_C cd ../.. # go to folders up # WD < you are here # └── TestFolder_A # └── TestFolder_B # └── TestFolder_C ## Remove test folders rmdir TestFolder_A # ✘ does not work because there are sub-folder inside TestFolder_A # ☛ rmdir deletes only empty folders # ✔︎ we have to delete subfolder by subfolder rmdir TestFolder_A/TestFolder_B/TestFolder_C # WD < you are still here but subfolder C is gone # └── TestFolder_A # └── TestFolder_B rmdir TestFolder_A/TestFolder_B # WD < you did not move but you deleted sub-folder B # └── TestFolder_A rmdir TestFolder_A/ # WD < you are still here and all sub-folders are gone
❖ Challenge #1: The command rmdir removed only empty folders. Can you find a way to remove all subfolders at once? Tip: Use the manual page to get help.
Suggestion #1
We can use the remove rm command with the recursive option.
Careful with the force options - there is no undo option!
mkdir -p TestFolder_X/TestFolder_XX/TestFolder_XXX
rm -fr TestFolder_X/TestFolder_XX/TestFolder_XXX
# ➜ Option -f, --force: ignore nonexistent files and arguments, never prompt
# ➜ Option -r, -R, --recursive: remove directories and their contents recursively
❖ Challenge #2: Does it matter is you use capital letters or not? Find a way to test if your local terminal is case sensitive.
Suggestion #2
Some filesystems are case insensitive. It is important to know if TEST and test is the same on your computer.
mkdir TEST
mkdir test
# mkdir: TEST: File exists ➜ Terminal is not case sensitive
Data Streams¶
Every program we run on the command line has three data streams. We can alter these data streams in interesting and useful ways.
STDIN (0) - Standard input STDOUT (1) - Standard output (by default printed to the terminal) STDERR (2) - Standard error (by default printed to the terminal)
Redirect Output (overwrite)¶
We change the default of STDOUT / STDERR from terminal to file.
## Print message on terminal echo "Hello Terminal" # works and prints result to terminal icho "Hello Terminal" # typo, does not work and prints error to terminal ## Redirect outputs (STDOUT and STDERR) in files echo "Hello Terminal" 1> test1.txt 2> errors.txt more test1.txt # you could also use [less] or [cat] instead of [more] more errors.txt # empty because there was no error ## Redirect outputs (STDOUT and STDERR) in files icho "Hello Terminal" 1> test1.txt 2> errors.txt cat test1.txt # empty because there was a typo in the command cat errors.txt # error message ## Redirect outputs (STDOUT and STDERR) in one file echo "Hello Terminal" > test1.txt 2>&1 cat test1.txt ## Redirect only output (STDOUT) in a file echo "Hello Terminal" > test1.txt # you do not need 1> if you only print STDOUT cat test1.txt
Careful, you overwrite the file if you redirect STDOUT and STDERR into a file with the same name.
## Overwrite the previous message echo "Nothing in life is to be feared." > text1.txt more text1.txt
You can print to different files and combine (concatenate) the files.
## Print another message to a different file echo "It is only to be understood." > text2.txt ## Merge content of files cat text1.txt text2.txt > text12.txt cat text12.txt
Redirect Output (append)¶
There is an alternative to merge mulitple output files. We can use the double greater than operator (>>) to appended the output to an existing file.
## Add (>>) a third line to the combined file echo "Marie Curie" >> text12.txt cat text12.txt
Redirecting from a File¶
We can use the less than operator (<) to
wc -l text12.txt wc -l < text12.txt
It looks similar but there is a subtle difference.
wc -l text12.txt > count.txt wc -l < text12.txt >> count.txt cat count.txt # 3 text12.txt # 3 <file name is missing>
When we redirect the STDIN we send the data "anonymously". The program does not know where the data is coming from. A trick to avoid unwanted ancillary information.
## This is ugly wc -l text12.txt > count.txt echo "We have $(cat count.txt) lines." ## This is better wc -l < text12.txt > count.txt echo "We have $(cat count.txt) lines."
Piping¶
Sending data from one program (STDOUT) to another one (STDIN) is called piping. We us the vertical bar | to feed the output from one cammand to the next.
## Create a multi-line message echo -e "Think Like a Proton\nStay Positive" > proton.txt # ➜ \n stands for newline and divides the string into two lines cat proton.txt ## Copy first / last line to a new file cat proton.txt | head -n 1 cat proton.txt | tail -n 1
Log-Files¶
You can use the redirect option to create log-files of e.g. your terminal sessions.
## Create an empty file and fill it rm ZERO.txt; touch ZERO.txt echo "Test Log File" >> ZERO.txt echo "-------------------------------" >> ZERO.txt echo "My working directory:${PWD}" >> ZERO.txt echo "My tree version:" >> ZERO.txt tree --version | cut -c1-12 >> ZERO.txt echo "-------------------------------" >> ZERO.txt tree >> ZERO.txt echo "-------------------------------" >> ZERO.txt date +"%d/%m/%y" >> ZERO.txt echo "-------------------------------" >> ZERO.txt clear; cat ZERO.txt
❖ Challenge #3: Let us create a virtual dice and safe the results of three throws in a text file.
For this purpose we use the internal Bash function $RANDOM. It returns a (pseudo)random integer.
## Default use echo ${RANDOM} # > The default range is too large (0 - 32767) ## Restrict the range echo $(( RANDOM % 7)) # > Better but we need a range from 1-6 ## Range between 1-6 echo $((1 + RANDOM % 6))
So far so good. Now, we need to roll our dice three times and safe the results in a file.
Suggestion #3
3.1 Step-by-Step
echo $((1 + RANDOM % 6)) > random_number_1.tmp
echo $((1 + RANDOM % 6)) > random_number_2.tmp
echo $((1 + RANDOM % 6)) > random_number_3.tmp
cat random_number_[123].tmp > random_numbers_S1.txt
rm *.tmp
cat random_numbers_S1.txt
echo $((1 + RANDOM % 6)) > random_numbers_2.txt
echo $((1 + RANDOM % 6)) >> random_numbers_2.txt
echo $((1 + RANDOM % 6)) >> random_numbers_2.txt
cat random_numbers_S2.txt
for i in 1 2 3
do
echo "Random Number ${i}: ${RANDOM:3}"
done > random_numbers.txt
cat random_numbers_S3.txt
Copy, Rename and Remove¶
## Copy a file - original is kept cp text12.txt Marie_Curie.txt ls -l cat text12.txt Marie_Curie.txt ## Rename (move) file - original is lost mv ZERO.txt logfile.txt ls -l ## Remove file(s) rm text12.txt text.txt ls -l
❖ Challenge #4.1: What is the difference between the two commands.
cp file.txt newfile.txt cat file.txt > newfile.txt
Suggestion #4.1
- The first line creates a copy of the file. It is possible to copy any file.
cp file.pdf newfile.pdf # ✔︎
cat. For example, it would not work for PDFs.
cat file.pdf > newfile.pdf # ✖︎
❖ Challenge #4.2: When would you use mv instead of cp?
mv file.txt newfile.txt
Suggestion #4.2
Use the move command to rename and/or move a file. Copy a file is safer because you keep the original but large files might take a while to copy and use disk space.
Wildcards¶
In the previous chapter you created various text files. You can list all files in your working directory or select only specific files. Wildcards can be very hand for this task.
## List all text files ls *.txt # list all files with ending .txt ls text?.txt # list all files starting with test, followed by one character, and ending with .txt ls text[123].txt # list all files starting with test, followed by 1,2 or 3, and ending with .txt # ➜ * any characters # ➜ ? one charachter # ➜ [123] a group - meaning 1, 2, or 3 ## Remove multiple files rm text1.txt text2.txt text3.txt rm text[123].txt
❖ Challenge #5.1: Can you find a command line to delete the index (I1 or I2) samples but keep the forward (R1) and reverse (R2) reads?
Sample_GX0I1_R1.fq.gz Sample_GX0I1_R2.fq.gz Sample_GX0I1_I1.fq.gz Sample_GX0I1_I2.fq.gz Sample_GX0I2_R1.fq.gz Sample_GX0I2_R2.fq.gz Sample_GX0I2_I1.fq.gz Sample_GX0I2_I2.fq.gz
Suggestion #5.1
There are usually more than just one possible solution. Some might be better (e.g. faster, more secure) than others but it is paramount you understand what you do.
## Suggestion 5.1a
rm -i Sample_GX0I1_I1.fq.gz Sample_GX0I1_I2.fq.gz Sample_GX0I2_I1.fq.gz Sample_GX0I2_I2.fq.gz
# ➜ Safe and it works but imagine you have a few hundred files.
## Suggestion 5.1b
rm -i Sample_GX0I?_I?.fq.gz
# ➜ Also safe and would work just fine as long as all samples follow the same name structure.
## Suggestion 5.1c
rm -i *_I1.*
# ➜ Short and precise but can be dangerous.
## Tip: You might test your wildcards first?
ls -ah *_I1.*
❖ Challenge #5.2: What is the problem with the following command? Can you correct it?
# cat sequence*.fa >> sequence_all.fa # ✖︎✖︎✖︎ Do not use!
Suggestion #5.2
The command will never finish until your hard drive is filled. The wildcard also includes the output file and this would create a "never ending" circle. Better/correct solutions would include:
cat sequence*.fa >> all_sequence.fa
cat sequence*.fa >> different_path/sequence_all.fa
Terminal History¶
You might be familiar with the history of your internet browser. The terminal has a history too. This is great because with command history we cannot only search the past but it also means not to retype previous commands. Use arrow up and down to travel within your history. You can also access it:
history
With no options (default), you will see a list of previous commands with line numbers. Lines prefixed with a ‘*’ have been modified. An argument of n lists only the last n lines.
-c clear history -d offset Delete the history entry at position offset. -d start-end Delete the history entries between positions start and end -a Append the new history lines to the history file. -n Append the history lines not already read from the history file to the current history list. -r Read the history file and append its contents to the history list. -w Write out the current history list to the history file.
Syntax examples:
history [n] history -c history -d offset history -d start-end history [-anrw] [filename]
❖ Challenge #6: Why would you need a history of your commands?
Suggestion #6
There are many good reason. Let me list a few, for me obvious ones. I am happy to learn new ones if you like to share.
- You might have noticed that with the
- You can also used the history for troubleshooting or to keep a log file of your session.
Some Sequence Examples¶
# Download a sequence fasta file pwd # make sure this is the right place for the download curl -O http://gdc-docs.ethz.ch/GeneticDiversityAnalysis/GDA20/data/RDP_16S_Archaea_Subset.fasta ls -lh RDP_16S_Archaea_Subset.fasta # Count the number of lines in the file wc -l RDP_16S_Archaea_Subset.fasta # Have a look at the fasta file less RDP_16S_Archaea_Subset.fasta # Have a look at the first 15 lines head -n 15 RDP_16S_Archaea_Subset.fasta # Count the number of sequences grep ">" -c RDP_16S_Archaea_Subset.fasta # Find a specific motif and highlight it grep "cggattagatacccg" --color RDP_16S_Archaea_Subset.fasta # Count how many time you found the motif grep "cggattagatacccg" -c RDP_16S_Archaea_Subset.fasta # Find alternatives step-by-step grep "cgggaggc" -c RDP_16S_Archaea_Subset.fasta grep "cgggtggc" -c RDP_16S_Archaea_Subset.fasta grep "cgggcggc" -c RDP_16S_Archaea_Subset.fasta grep "cggggggc" -c RDP_16S_Archaea_Subset.fasta # Find alternative faster grep "cggg[atcg]ggc" -c RDP_16S_Archaea_Subset.fasta
Your Turn¶
1 - Create a text file with a title and your username.
2 - Add the last 20 command lines you used.
3 - Add a date to the bottom of the file.
Suggestion
echo "=== Safe My History ===" > MyHistory.txt
echo "${USER}" >> MyHistory.txt
echo "-----------------------" >> MyHistory.txt
history | tail -n 20 >> MyHistory.txt
echo "-----------------------" >> MyHistory.txt
date "+%A, %d.%B %Y" >> MyHistory.txt
clear; cat Chapter8.txt
