Commands
Find below the most important commands for working on Euler. A collection of important Linux bash commands could be found here. Tutorials by Ryan Chadwick on Linux and bash scripting are very useful.
Job related
Submit a job
sbatch < submit.tool.slurm.sh
Overview of the submitted jobs (pending and running)
squeue --format='%A %i %j %P %4T %C %m %M %R' | column -t
The Job-ID and the Array-ID is normally different.
Kill specific job
scancel <Job-ID>/<Array-ID>
Kill all running jobs
scancel --user=$USER
Kill job 15-23 of Array-ID
scancel <Array-ID>_[15-23]
CPU and memory usage of running jobs
myjobs -r
myjobs -j <Job-ID>/<Array-ID>
Efficiency of finished job(s)
source /cluster/project/gdc/shared/stack/GDCstack.sh
module load reportseff/2.7.6
reportseff --format JobID,ReqCPUS,CPUEff,ReqMem,MaxRSS,MemEff,Timelimit,elapsed,TimeEff <Job-ID>/<Array-ID>
Or the efficency of all jobs in the last 24 hours.
reportseff --user $USER --since d=1
Get a summary of your ressource usage the last couple of days
/cluster/software/bin/get_inefficient_jobs
Connect to a node to check on a job. This is an advanced command. Can be useful to check on the real-time CPU usage or data in the local scratch space.
srun --interactive --jobid <job-ID> --pty bash
Software stack
Load GDC software stack
source /cluster/project/gdc/shared/stack/GDCstack.sh
View all available tools
module --show-hidden avail
Search for tool
module --show-hidden avail toolXY
Unload all modules
module purge
Data Management
Disk usage of your personal home and your scratch
lquota
Provide the size of the folder "mapping"
du -sh --si mapping
Count number of files and directories in "raw_data".
find raw_data | wc -l
Archive folder data1
tar cvzf data1.tar.gz data1
#delete the folder
rm -rf data1
Extract archive data1.tar.gz
#get overview about the archive
tar -ztvf data1.tar.gz
#extract entire archive
tar xvf data1.tar.gz
#extract specific folder of the archive
tar xvf data1.tar.gz data1/raw
#Generate list with md5sums
md5sum *fq.gz > md5sums.txt
#Verify md5sums e.g. on another Server
md5sum --check md5sums.txt
/cluster/apps/local/lquota /cluster/work/gdc
Additional documentation
Official SLURM documentation. Keep in mind that Slurm is not Slurm and every cluster has own configurations.