Skip to content

High-performance computing (HPC) clusters are an essential part of modern bioinformatics. Many of the analyses covered in this course, such as genome assembly, sequence alignment, and variant calling, involve datasets that are simply too large and too computationally demanding to run on a personal laptop. This page introduces the key concepts behind HPC systems so that you can use them confidently and responsibly.

For practical exercises and ETH-specific instructions, please refer to the Euler Tutorial.

What Is an HPC Cluster?

An HPC cluster is a collection of many computers, called nodes, that are connected by a high-speed network and managed as a single system. Rather than running jobs on your own laptop, you submit them to the cluster, which assigns the necessary computing resources and runs the job on your behalf.

A typical HPC cluster consists of several types of nodes:

  • Login nodes: The entry point to the cluster. You connect here via SSH, prepare your data, write scripts and submit jobs. Login nodes are shared by all users and should never be used for heavy computation.
  • Compute nodes: The workhorses of the cluster. These are where your jobs actually run. They are allocated by the job scheduler and dedicated to your job for its duration.
  • Storage nodes: Manage the file systems shared across the cluster, including your home directory and any scratch space.

Why Use an HPC Cluster?

Personal laptops are well suited to small tasks, but they have real limitations when it comes to bioinformatics:

  • Memory: Assembling a genome or loading a large alignment file can require hundreds of gigabytes of RAM, far beyond what a laptop can provide.
  • Processing power: Many bioinformatics tools can parallelise their work across many CPU cores simultaneously. A cluster may have hundreds or thousands of cores available.
  • Storage: Raw sequencing data can occupy terabytes of disk space. Clusters provide shared, backed-up storage at a scale that personal computers cannot match.
  • Runtime: Some analyses take days to complete. Running them on a cluster means your laptop remains free and the job continues even if you close your connection.

The Job Scheduler

The job scheduler is the software that manages access to the cluster's resources. When many users are submitting jobs at the same time, the scheduler decides which jobs run when, and on which nodes, based on the resources requested and the current load on the system.

The most widely used job scheduler in academic HPC environments is SLURM (Simple Linux Utility for Resource Management). Other schedulers such as PBS, LSF and SGE exist, but SLURM has become the standard at most European universities and research institutions including ETH Zurich.

To run a job on the cluster, you write a job script: a plain text file that contains both the resource requirements for your job (how many CPUs, how much memory, how long it will run) and the commands you want to execute. You then submit this script to the scheduler using the sbatch command.

sbatch my_job_script.sh

The scheduler places your job in a queue and runs it as soon as the requested resources become available. While your job is waiting or running, you can check its status:

squeue --me       # show your jobs in the queue
sacct -j <jobid>  # show details of a completed job
scancel <jobid>   # cancel a running or pending job

Resource Requests

One of the most important skills when using an HPC cluster is requesting the right amount of resources. Requesting too little can cause your job to fail. Requesting too much wastes shared resources and pushes your job further back in the queue.

A typical SLURM job script header looks like this:

#!/bin/bash
#SBATCH --job-name=my_analysis
#SBATCH --time=04:00:00          # maximum run time (HH:MM:SS)
#SBATCH --mem=16G                # memory per node
#SBATCH --cpus-per-task=4        # number of CPU cores
#SBATCH --output=job_%j.log      # log file (%j = job ID)

Good practice is to start with a small test run on a subset of your data to get a rough idea of the time and memory your job needs, and then scale up accordingly.

The Module System

HPC clusters typically use a module system to manage software. Rather than installing tools globally, software is made available in self-contained modules that users can load and unload as needed. This allows multiple versions of the same tool to coexist on the cluster without conflict.

module avail             # list all available modules
module load bwa/0.7.17   # load a specific version of a tool
module list              # show currently loaded modules
module unload bwa        # unload a module
module purge             # unload all modules

Always specify the version of a module when loading it. This ensures your analysis is reproducible and will not be affected by future updates to the default version.

Storage and Data Management

HPC clusters typically offer several types of storage, each with different purposes:

  • Home directory: A small, backed-up space for scripts, configuration files and important results. Not suitable for large datasets.
  • Scratch space: A large, fast, temporary storage area for active computations. Files here are not backed up and are typically deleted after a defined period. Use scratch space for intermediate files and large input data during a job.
  • Project or group storage: Shared space for a research group, often larger and with longer retention than scratch.

Do not store important data on scratch

Scratch space is designed for temporary use. Always copy important results to your home directory or a backed-up location before they are automatically deleted.

Parallelism: Cores, Threads and Tasks

Understanding the difference between parallelism concepts helps you write better job scripts and use cluster resources efficiently.

  • A CPU core is a single processing unit. Modern computers and cluster nodes have many cores.
  • Threads are lightweight units of execution within a single programme. Many bioinformatics tools (e.g. bwa, samtools, STAR) support multithreading, meaning they can use multiple cores simultaneously. You control this with a tool-specific option (often -t or --threads).
  • Tasks in SLURM refer to independent processes. For most single-tool bioinformatics jobs, you will run one task with multiple threads.

If a tool supports 4 threads, request 4 CPUs in your job script and pass --threads 4 to the tool. Always match the two — requesting 16 CPUs but only using 4 wastes resources.

Good Practices on a Shared System

An HPC cluster is a shared resource used by many researchers simultaneously. Using it responsibly benefits everyone.

Cluster Etiquette

  • Never run computationally intensive jobs on the login node. Use the scheduler.
  • Do not request more resources than you actually need.
  • Clean up temporary files and scratch space after your job completes.
  • Test your scripts on small data before submitting large jobs.
  • If something goes wrong, check the log files before contacting support.
  • Cite the cluster in your publications if required by your institution.

Further Reading

For practical exercises, example job scripts, and ETH Zurich specific instructions for the Euler cluster, please refer to the Euler Tutorial.