This page contains a collection of small but important topics that don't fit neatly into other sections. Consider it a reference page for quick access to key information. If you think anything is missing, feel free to let us know!
File Naming
You can name your files and folders anything you like. However, following naming conventions helps maintain a tidy, efficient, and easily navigable digital environment, reducing the risk of misplaced files and making it easier for everyone to find and manage documents.
General Rules and Best Practices
- Meaningful but short and descriptive
- Use consistent naming conventions
- Avoid special characters
- Use leading zeros for sorting
- Indicate versions and revisions
- Consider case sensitivity
- Avoid redundancy
Balance length and clarity
Bad: Genetic Diversity Analysis - Rolling Dice Exercise.txt
Better: GDA_RollingDice_BobaFett_V01.R
Naming convention and numerical order
Bad: water_sample_river_12.fa
Better: RW-012.fa
No spaces
# Does not work
echo "Test" > My First File
# Works but it is not a good idea
echo "Test" > "My First File"
# Also works but still not a good idea
echo "Test" > My\ First\ Test
# What you should use instead
echo "Test" > MyFirstTest.txt # camel case
echo "Test" > my_first_test.txt # snake case
The same applies to folders:
# Bad idea
mkdir "Material and Methods"
# Does not work
cd Materials and Methods
# Works but is not optimal
cd Materials\ and\ Methods
cd "Material and Methods"
# Prefer underscores over spaces
mkdir Material_and_Methods
cd Material_and_Methods
Your turn. Find better solutions for the following file names:
- Spacing and case sensitivity
my new file
a) mynewfile.txt (simple but difficult to read)
b) MyNewFile.txt (simple and easier to read, camel case)
c) My_New_File.txt (clear but needs more space, snake case)
Note: Avoid spaces and use extensions to indicate the file format. Be aware of case sensitivity.
- Numbers
Sample1 (ph5) temperatur high.fasta
Avoid special characters and keep it short. If you have numbered files, use leading zeros to keep the order when sorting. Sort works character by character and numbers above 9 are not recognised without leading zeros.
Better:
To fix inconsistent numbering in existing file names:
- Special characters
GDA Final Report: Irène Müller.md
You might be Irène Müller in life, but on the terminal you need to settle for Irene_Mueller. It is nothing personal.
- What about dots?
zip file.zip primer*.fq
Use dot-based filename extensions to identify file types. The file type should not be lost in the archive name.
- Keep it short
# Messy looking sample list
sample1_rive_SG.fa
sample02_lake_SG.fasta
s3_water-lake_Zurich.fa
sample4_water_lake_ZH.fasta
sam5_water-river_TG.fa
# Clean and consistent sample list
S01_R_SG.fa
S02_L_SG.fa
S03_L_ZH.fa
S04_L_ZH.fa
S05_R_TG.fa
File Extensions
File extensions are critical for both users and systems, enabling efficient file management, ensuring compatibility and facilitating automated processes. It is important to specify the file type correctly.
# Text file
echo "Test" > file.txt
# Temporary file
echo "Temp-Test" > file.tmp
# Sequence file
echo ">Seq01\nATGCCGTCACCGT" > file.fa
File Formats
FASTA
The FASTA format is a text-based format for representing one or more nucleotide or protein sequences. The first line (header) of a FASTA record starts with a greater-than symbol, followed by a unique description of the sequence (e.g. an accession number). The header is followed by the actual sequence as a string, spread over one or more lines.
Multiple-line (wrapped) FASTA file:
>Sequence_001
ATGGGCGTCACGTCCACGTTCACCGTGTTA
TGGAATGCGTCACGTCAACTGGGGT
>Sequence_002
ATGGCCGTCACGTCCACGTTCACCGTGTTATGGAATGCGACACGTCAACTGGGGT
>Sequence_003
ATGGCCGTCACGTCCACGTT
Single-line FASTA file:
>Sequence_001
ATGGGCGTCACGTCCACGTTCACCGTGTTATGGAATGCGTCACGTCAACTGGGGT
>Sequence_002
ATGGCCGTCACGTCCACGTTCACCGTGTTATGGAATGCGACACGTCAACTGGGGT
>Sequence_003
ATGGCCGTCACGTCCACGTT
To count the number of sequences in a FASTA file, count all lines beginning with a > sign:
0.5 * n(lines) is not n(sequences)
The number of lines divided by two is not necessarily the number of sequences in a multi-sequence FASTA file, because sequences can span multiple lines.
FASTQ
A FASTQ file is a text-based format for storing both a biological sequence (usually nucleotide sequences) and its associated quality scores.
Each entry in a FASTQ file consists of four lines:
1: @M01072:41:000000000-A942B:1:1101:11853:2457 1:N:0:1
2: GTGCCAGCAGCCGCGGTAATACGTAGGTGGCAAGCGTTATCCGGATTTATTGTGCGTAAAGGGAACGC...
3: +
4: >>1>>11>11>>1EC?E?CFBFAGFC0GB/CG1EACFE/BFE///AEG1DF122A/B///21//0BEG...
Line 1: Header line with a unique sequence identifier.
Line 2: The nucleotide sequence (A, C, T, G and N).
Line 3: A separator, which is simply a plus (+) sign.
Line 4: Base call quality scores, Phred +33 encoded using ASCII characters.
To count the number of sequences in a FASTQ file, counting lines with @ may not work because the quality line can also contain @ characters. The safest approach is to use the fact that every record is exactly four lines:
## Number of lines divided by 4
echo $(($(wc -l < file.fq) / 4))
## Lines starting and ending with + (not bulletproof)
zgrep -c "^+$" file.fq.gz
0.25 * n(lines) = n(sequences)
The number of lines divided by four equals the number of sequences in a FASTQ file.
FASTG
FASTG is a file format for the faithful representation of genome assemblies. The G stands for graph. While FASTA files represent genomes as linear sequences, many genomes contain polymorphisms that cannot be described linearly, and most assemblies contain errors and problematic regions. FASTG addresses this with a flexible, graph-based approach to encoding sequence variability. Bandage is a program for visualising de novo assembly graphs using FASTG input files.
Example from the Megahit metagenome assembler:
>NODE_12427_length_4458_cov_84.0000_ID_24853':NODE_164443_length_1445_cov_44.3323_ID_328885';
CGCTTGCTCGATTTGACCCCATAAGCGGAATCACCACCCAATAGCATGTTGCCGCCAGAAGCAGAACCCC...
>NODE_515499_length_44875_cov_57.9870_ID_1030997:NODE_12427_length_4458_cov_84.0000_ID_24853;
CTTTGCTTTTGTAATGGTGCCTTGCAATAAATTCTACAAGAGCTAAAAAAGAGTCTCATCATTCACGAC...
FAST5 / POD5
A FAST5 file is a hierarchical data format (HDF5) used to store raw signal data and metadata from Oxford Nanopore Technologies (ONT) sequencing devices.
POD5 is a newer format developed by ONT to replace FAST5. It reads and writes data faster, uses less computing power, and produces smaller raw data files. Raw current signal data from an ONT device are typically base-called into FASTQ sequence reads.
SAM / BAM
SAM (Sequence Alignment/Map) and BAM (Binary Alignment/Map) are file formats for storing DNA sequence alignment data.
SAM files are plain text files containing a header section and an alignment section. The header provides information about the reference genome, sequence dictionary, and other metadata. The alignment section contains records for each aligned read, including the read ID, reference sequence name, alignment position, quality scores, flags, and more.
@HD VN:1.6 SO:coordinate
@SQ SN:chr1 LN:248956422
@RG ID:sample1 LB:lib1 SM:sample1 PL:illumina
@PG ID:bwa PN:bwa VN:0.7.17-r1188 CL:bwa mem -t 4 ref.fa read1.fq read2.fq
read1 99 chr1 1000 30 10M1D20M = 2000 0 AGCTTAGCTAGCTAGCTAGCTAGCTAGCTA IIIIIIIIIIIIIIIIIIIIIIIIIIIIIII
read2 147 chr1 2000 30 30M = 1000 0 CAGCTAGCTAGCTAGCTAGCTAGCTAGCTA IIIIIIIIIIIIIIIIIIIIIIIIIIIIIII
@HD: header information including version and sort order.@SQ: reference sequences (chromosomes) with names and lengths.@RG: read group information including library, sample and platform.@PG: program used for alignment with version and command-line parameters.- Alignment records contain read ID, flags, reference name, position, mapping quality, CIGAR string, mate information, and tags with additional details.
BAM files are binary, compressed versions of SAM files. They are indexed, which makes them more efficient in terms of storage and retrieval speed.
VCF
A VCF (Variant Call Format) file is a standard format for storing genetic variations such as SNPs, insertions, deletions, and structural variants. Each record includes the chromosome location, reference allele, alternative allele(s), quality scores, annotations, and genotype information for each sample.
##fileformat=VCFv4.3
##fileDate=2023-06-28
##source=GenomeAnalyzerV2.0
##reference=GRCh38
##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
#CHROM POS ID REF ALT QUAL FILTER INFO FORMAT SAMPLE1 SAMPLE2 SAMPLE3
1 100 rs123 A T 100 PASS . GT 0/1 1/1 0/0
1 200 rs456 G C 50 PASS . GT 1/1 0/0 1/1
1 300 rs789 C T 80 PASS . GT 1/1 1/1 0/1
- Lines starting with
##are metadata lines. - The
#CHROMline contains the column headers. -
Each subsequent line represents a single variant record. The genotype field (GT) encodes whether each sample carries zero, one, or two copies of the alternative allele (0/0, 0/1, 1/1).
Encoding
A code is a system of rules for translating information. Character encoding in textual data represents a repertoire of characters as standardised sequences. A familiar example is Morse code:
--. -.. -.-. (GDC)
ASCII and Unicode (UTF) are character encoding standards for electronic communications. ASCII is suitable for basic text and control codes, while Unicode provides extensive support for global text representation.
| Binary code | ASCII code | Letter |
|---|---|---|
| 01000111 | (64 + 4 + 2 + 1 =) 071 | G |
| 01000100 | (64 + 4 =) 068 | D |
| 01000011 | (64 + 2 + 1 =) 067 | C |
Plain text files need the correct encoding to be decoded properly. Use the file command to determine the file type and encoding:
A common source of problems is the newline (end of line) encoding, which differs between operating systems:
| OS | EOL | Encoding |
|---|---|---|
| Linux | LF | \n |
| macOS | LF | \n |
| MS Windows | CRLF | \r\n |
A friendly tip
Having trouble importing files? Double-check the file encoding. Different systems and programs use various encodings such as UTF-8 or ISO-8859-1, and mismatches can cause errors. If you are unsure, open the file in a text editor to confirm the encoding.
K-mers
In bioinformatics, K-mers are substrings of length k from biological sequences. The abundance of k-mers in a genome, genomic region, or class of sequences can serve as a signature of the underlying sequence. Comparing k-mer abundances is computationally simpler than sequence alignment and is an important method in non-alignment sequence analysis. It is also commonly used as the first stage of analysis before alignment.
Example: k-mers for the DNA string GTC
| k | k-mer(s) |
|---|---|
| 1 | G, T, C |
| 2 | GT, TC |
| 3 | GTC |
Compressing Sequence Files
Sequencing centres provide compressed sequence files for good reasons. Compressed files are smaller and more secure to download. Keep your data compressed, as many applications can handle compressed files directly. In general, compress both FASTQ and FASTA files to save disk space and for safer data transfer.
## Working directory
mkdir -p ${HOME}/MDA/qc
cd ${HOME}/MDA/qc
## Download a paired-end toy dataset
curl -O "https://www.gdc-docs.ethz.ch/GeneticDiversityAnalysis/GDA/data/C-1_R1.fastq.gz"
curl -O "https://www.gdc-docs.ethz.ch/GeneticDiversityAnalysis/GDA/data/C-1_R2.fastq.gz"
wget or curl?
Both wget and curl are command-line tools for transferring data. wget is focused on downloading files from the web and provides detailed progress information. curl supports a wider range of protocols and is more flexible for complex tasks.
The zcat command allows you to view the contents of a gzipped file without extracting it:
Shell differences
zcat behaves differently in bash and zsh. On macOS, use gzcat or redirect with <:
Random sampling
head -n 40 takes only the top sequences, not a random subset. A better alternative is seqtk:
seqtk sample -s1505 C-1_R1.fastq.gz 10 > random_subset_C-1_R1.fq
seqtk sample -s1505 C-1_R2.fastq.gz 10 > random_subset_C-1_R2.fq
Use the same seed (-s) to keep paired reads in sync.
GNU Zip
## Compress
gzip subset_C-1_R*.fq
## Compression info
file subset_C-1_R*.fq.gz
gzip -l subset_C-1_R*.fq.gz
## Decompress
gunzip subset_C-1_R*.fq.gz
## Compression levels (1 = fast, 9 = best compression, default = 6)
time gzip -1 test.fq # fast, lower compression
time gzip -6 test.fq # default, good balance
time gzip -9 test.fq # slow, highest compression
man gzip
gzip --help
TAR Archive
## Create an uncompressed archive
tar cfv subset_C-1_R1R2.fq.tar subset_C-1_R1.fq subset_C-1_R2.fq
## Extract
tar xfv subset_C-1_R1R2.fq.tar
## Create a compressed archive
tar cfvz subset_C-1_R1R2.fq.tar.gz subset_C-1_R1.fq subset_C-1_R2.fq
## Extract compressed archive
tar xfvz subset_C-1_R1R2.fq.tar.gz
man tar
Exploring Without Decompressing
You do not always need to fully decompress a file to check what is inside.
## View the first four lines of a gzipped file
zcat C-1_R1.fastq.gz | head -n 4
## Alternative using gzip flags
gzip -cd C-1_R1.fastq.gz | head -n 4
## List contents of a tar.gz archive without unpacking
tar -tf C-1_R1.fastq.tar.gz
## Detailed information about a zip file
zipinfo file.zip
Watch Disk Space
Before unpacking files, make sure there is enough disk space available (df -h). Consider using scratch space that is not backed up. Compress large files and archive small files to keep your project folder manageable.
File Integrity
Every time you transfer data, there is a chance that a file will be corrupted. To verify the integrity of a file, use md5sum, which calculates a checksum fingerprint for a file.
md5sum C-1_R[12].fastq.gz
# 4a2e8876742302fad5bd24cba76c3cc6 C-1_R1.fastq.gz
# e0ce4ae266e8b2cedbc9580a3025712a C-1_R2.fastq.gz
Not working?
On macOS, try md5 instead: md5 C-1_R[12].fastq.gz
Renaming a file does not change the checksum. Only changing the file content does:
## Renaming does not change the checksum
cp C-1_R1.fastq.gz test.fq.gz
md5sum C-1_R1.fastq.gz test.fq.gz
# Both checksums are identical
## Recompressing at a different level changes the checksum
gunzip -c C-1_R1.fastq.gz > test.fq
gzip test.fq
md5sum C-1_R1.fastq.gz test.fq.gz
# Checksums differ because compression level differs
Data Transfer
Always compress or archive files before transferring them, and verify the checksum afterwards to confirm the transfer was successful.
