Parallel batch jobs¶
Computations involving a very large number of independent computations should be combined in some way to reduce the number of jobs submitted to Slurm. Here we illustrate one strategy for doing this using GNU Parallel and srun. The parallel program executes tasks simultaneously until all tasks have been completed.
Here’s an example script, parallel.sbatch
:
#!/bin/sh
#SBATCH --time=01:00:00
#SBATCH --partition=broadwl
#SBATCH --ntasks=28
#SBATCH --mem-per-cpu=2G # NOTE DO NOT USE THE --mem= OPTION
# Load the default version of GNU parallel.
module load parallel
# When running a large number of tasks simultaneously, it may be
# necessary to increase the user process limit.
ulimit -u 10000
# This specifies the options used to run srun. The "-N1 -n1" options are
# used to allocates a single core to each task.
srun="srun --exclusive -N1 -n1"
# This specifies the options used to run GNU parallel:
#
# --delay of 0.2 prevents overloading the controlling node.
#
# -j is the number of tasks run simultaneously.
#
# The combination of --joblog and --resume create a task log that
# can be used to monitor progress.
#
parallel="parallel --delay 0.2 -j $SLURM_NTASKS --joblog runtask.log --resume"
# Run a script, runtask.sh, using GNU parallel and srun. Parallel
# will run the runtask script for the numbers 1 through 128. To
# illustrate, the first job will run like this:
#
# srun --exclusive -N1 -n1 ./runtask.sh arg1:1 > runtask.1
#
$parallel "$srun ./runtask.sh arg1:{1} > runtask.sh.{1}" ::: {1..128}
# Note that if your program does not take any input, use the -n0 option to
# call the parallel command:
#
# $parallel -n0 "$srun ./run_noinput_task.sh > output.{1}" ::: {1..128}
In this example, our aim is to run script runtask.sh
128
times. The --ntasks
option is set to 28, so at most 28 tasks
can be run simultaneously.
Here is the runtask.sh
script that is run by GNU parallel:
#!/bin/sh
# This script outputs some useful information so we can see what parallel
# and srun are doing.
sleepsecs=$[ ( $RANDOM % 10 ) + 10 ]s
# $1 is arg1:{1} from GNU parallel.
#
# $PARALLEL_SEQ is a special variable from GNU parallel. It gives the
# number of the job in the sequence.
#
# Here we print the sleep time, host name, and the date and time.
echo task $1 seq:$PARALLEL_SEQ sleep:$sleepsecs host:$(hostname) date:$(date)
# Sleep a random amount of time.
sleep $sleepsecs
To submit this job, copy both parallel.sbatch
and
runtask.sh
to the same directory, and run chmod +x
runtask.sh
to make runtask.sh
executable. Then the job can be
submitted to the Slurm queue:
sbatch parallel.sbatch
When this job completes, you should see output files with names
runtask.sh.N
, where N
is a number between 1 and 128. The
content of the first output file (i.e., runtask.sh.1
) should look
something like this:
task arg1:1 seq:1 sleep:14s host:midway2-0002 date:Thu Jan 10 09:17:36 CST 2017
Another file runtask.log
is also created. It gives a list of
the completed jobs. (Note: If the sbatch is submitted again, nothing
will be run until runtask.log
is removed.)
It is also possible to use this same technique to run multithreaded
tasks in parallel. Here is an example sbatch script,
parallel-hybrid.sbatch
, that distributes multithreaded
computations (each using 28 CPUs) across 2 nodes:
#!/bin/sh
#SBATCH --partition=broadwl
#SBATCH --time=01:00:00
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=28
#SBATCH --exclusive
# Load the default version of GNU parallel.
module load parallel
srun="srun --exclusive -N1 -n1 --cpus-per-task $SLURM_CPUS_PER_TASK"
# Instead of $SLURM_NTASKS, use $SLURM_NNODES to determine how
# many jobs should be run simultaneously.
parallel="parallel --delay 0.2 -j $SLURM_NNODES --joblog runtask.log --resume"
# Run the parallel command.
$parallel "$srun ./runtask.sh arg1:{1} > runtask.sh.{1}" ::: {1..6}