Using Midway

Now that you are able to log in to Midway (Connecting to RCC Resources), upload and access your files on the cluster (Data Storage and Data Transfer), and load software tools using the module system (Software), you are ready for the next step: scheduling access to the RCC compute cluster to perform computations. This is the topic of this section of the RCC User Guide.

Overview

The Midway compute cluster is a shared resource used by the entire University of Chicago community. Sharing computational resources creates unique challenges:

  1. Jobs must be scheduled in a way that is fair to all users.
  2. Consumption of resources needs to be recorded.
  3. Access to resources needs to be controlled.

The Midway compute cluster uses a scheduler to manage requests for access to compute resources. These requests are called jobs. In particular, we use the Slurm resource manager to schedule jobs as well as interactive access to compute nodes.

Here, we give the essential information you need to know to start computing on Midway. For more detailed information on running specialized compute jobs, see Running Jobs on Midway.

Service Units and Allocations

Service Units (SUs) are a measure of the amount of computing resources consumed on a compute cluster. Computing resources in a compute cluster include processing units (also called CPUs or cores), memory, and Graphical Processing Units (GPUs). In standard settings, 1 SU equals usage of 1 processing unit for 1 hour, but the exact calculation will vary depending on the amount of memory requested, as well as additional factors like the use of GPUs and CPU architecture. The aim of the Service Unit (SU) is to provide a “fair” account of computing resources. For more information, please refer to the RCC Service Units webpage.

An “allocation” is a quantity of computing time (SUs) and storage resources that are granted to a group of users, usually a lab managed by a principal investigator (PI). Without an allocation, you cannot schedule and run jobs on the RCC compute cluster. For more information about SU allocations, see RCC Allocations.

Checking your account balance

The accounts tool can be used to check account balances. After logging into Midway, simply type:

$ accounts balance

If you are a member of multiple groups, this will display the allocations and usage for all your groups. The accounts balance command has a number of options for summarizing allocation usage. For information on these options, type

$ accounts balance --help

To see an overall summary of your usage, simply enter:

$ accounts usage

You can also get a more detailed breakdown of your usage by job using the --byjob option:

$ accounts usage --byjob

For more commands available in the accounts tool, type

$ accounts --help

Types of Compute Nodes

The Midway compute cluster is made up of compute nodes with a variety architectures and configurations. A partition is a collection of compute nodes that all have the same, or similar, architecture and configuration. Currently, Midway has the following partitions:

Cluster Partition Compute cores (CPUs) Memory Other configuration details
midway1 westmere 12 x Intel X5675 3.07 GHz 24 GB  
  sandyb 16 x Intel E5-2670 2.6GHz 32 GB  
  bigmem 16 x Intel E5-2670 2.6GHz 256 GB  
    32 x Intel E7-8837 2.67GHz 1 TB  
  gpu 16 x Intel E5-2670 2.6GHz 32 GB 2 x Nvidia M2090 or K20 GPU
    20 x Intel E5-2680v2 2.8GHz 64 GB 2 x Nvidia K40 GPU
  mic 16 x Intel E5-2670 2.6GHz 32 GB 2 x Intel Xeon Phi 5100 coprocessor
  amd 64 x AMD Opteron 6386 SE 256 GB  
  ivyb 20 x Intel E5-2680v2 2.8GHz 64 GB  
midway2 broadwl 28 x Intel E5-2680v4 2.4GHz 64 GB  
  bigmem2 28 x Intel E5-2680v4 @ 2.4 GHz 512 GB  
  gpu2 28 x Intel E5-2680v4 @ 2.4 GHz 64 GB 4 x Nvidia K80 GPU

You can also retrieve a summary of the partitions on Midway using the sinfo command:

$ sinfo -s

In the sinfo summary, the right-most number in the “NODES” column gives the total number of nodes in each partition. This summary also lists partitions that are reserved for use by certain labs.

Interactive Jobs

After submitting an “interactive job” on Midway, the Slurm job scheduler will connect you to a compute node, and will load up an interactive shell environment for you to use on that compute node. This interactive session will persist until you disconnect from the compute node, or until you reach the maximum requested time.

sinteractive

The command sinteractive is the recommended Slurm command for requesting an interactive session. As soon as the requested resources become available, sinteractive will do the following:

  1. Log in to the node.
  2. Change into the directory you were working in.
  3. Set up X11 forwarding for displaying graphics.
  4. Transfer your current shell environment, including any modules you have previously loaded.

To get started (with the default interactive settings), simply enter sinteractive in the command line:

$ sinteractive

By default, an interactive session times out after 2 hours. If you would like more than 2 hours, be sure to include a --time=HH:MM:SS flag to specify the necessary amount of time. For example, to request an interactive session for 6 hours, run the following command:

$ sinteractive --time=06:00:00

There are many additional options for the sinteractive command, including options to select the number of nodes, the number of cores per node, the amount of memory, and so on. For example, to request exclusive use of two compute nodes equipped with the Infiniband interconnect on the Midway1 sandyb partition for 8 hours, enter the following:

$ sinteractive --exclusive --partition=sandyb --nodes=2 --constraint=ib --time=08:00:00

For more details about these and other useful options, read below about the sbatch command, and see Running Jobs on Midway. Note that all options available in the sbatch command are also available for the sinteractive command.

Note

It is possible to request an interactive session on a Midway2 partition (e.g., broadwl) from a Midway1 login node (midway1.rcc.uchicago.edu), and vice versa. However, since access to files and software modules is different on Midway1 and Midway2, it is Best Practice to request Midway1 partitions from Midway1 login nodes, and Midway2 partitions from Midway2 login nodes.

srun

An alternative to the sinteractive command is the srun command:

$ srun --pty bash

Unlike sinteractive, this command does not set up X11 forwarding, which means you cannot display graphics using srun. Both the srun and sinteractive commands have the same command options.

Batch Jobs

The sbatch command is the command most commonly used by RCC users to request computing resources on the Midway cluster. Rather than specify all the options in the command line, users typically write an “sbatch script” that contains all the commands and parameters neccessary to run the program on the cluster.

In an sbatch script, all Slurm parameters are declared with #SBATCH, followed by additional definitions.

Here is an example of an sbatch script:

#!/bin/bash
#SBATCH --job-name=example_sbatch
#SBATCH --output=example_sbatch.out
#SBATCH --error=example_sbatch.err
#SBATCH --time=00:05:00
#SBATCH --partition=broadwl
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=16
#SBATCH --mem-per-cpu=2000

module load openmpi
mpirun ./hello-mpi

Here is an explanation of what each of these options does:

In this example, we have requested 4 compute nodes with 16 CPUs each. Therefore, we have requested a total of 64 CPUs for running our program. The last two lines of the script load the openMPI module and launch the MPI-based executable that we have called hello-mpi (see MPI Jobs).

Note

If the --partition option is not specified, the request will automatically be directed to the sandyb partition on Midway1. Since this is the default partition, it can be more heavily requested than other partitions. Therefore, it is often beneficial to override the default and select a different partition.

Continuing the example above, suppose that this script is saved in the current directory into a file called example.sbatch. This script is submitted to the cluster using the following command:

$ sbatch ./example.sbatch

Many other options are available for submitting jobs using the sbatch command. For more specialized computational needs, see Running Jobs on Midway. Additionally, for a complete list of the available options, see the Official SBATCH Documentation.

Note

When writing your sbatch script, make sure that it accesses only files that are available on the partition you are using. For example, if you plan to use the Midway2 Broadwell nodes (--partition=broadwl), then any files stored on /scratch/midway will be unavailable to your sbatch job; use /scratch/midway2 instead. Also note that none of the Midway2 partitions have local file systems, so you should not attempt to save files to /local/scratch. For more details on which files are accesible from Midway1 and Midway2, see Data Storage.

Managing Jobs

The Slurm job scheduler provides several command-line tools for checking on the status of your jobs and for managing them. For a complete list of Slurm commands, see the Slurm man pages. Here are a few commands that you may find particularly useful:

  • squeue: find out the status of jobs submitted by you and other users.
  • sacct: retrieve job history and statistics about past jobs.
  • scancel: cancel jobs you have submitted.

In the next couple sections we explain how to use squeue to find out the status of your submitted jobs, and scancel to cancel jobs in the queue.

Checking your jobs

Use the squeue command to check on the status of your jobs, and other jobs running on Midway. The simplest invocation lists all jobs that are currently running or waiting in the job queue (“pending”), along with details about each job such as the job id and the number of nodes requested:

$ squeue

Any job with 0:00 under the TIME column is a job that is still waiting in the queue.

To view only the jobs that you have submitted, use the --user flag

$ squeue --user=<username>

This command has many other useful options for querying the status of the queue and getting information about individual jobs. For example, to get information about all jobs that are waiting to run on the ivyb partition, enter:

$ squeue --state=PENDING --partition=ivyb

Alternatively, to get information about all your jobs that are running on the ivyb partition, type:

$ squeue --state=RUNNING --partition=ivyb

The last column of the output tells us which nodes are allocated for each job. For example, if it shows midway572 for one of the jobs under your name, you may type ssh midway572 to log in to that compute node and inspect the progress of your computation locally.

For more information, consult the command-line help by typing squeue --help, or visit the official online documentation.

Canceling your jobs

To cancel a job you have submitted, use the scancel command. This requires you to specify the id of the job you wish to cancel. For example, to cancel a job with id 8885128, do the following:

$ scancel 8885128

If you are unsure what is the id of the job you would like to cancel, see the JOBID column from running squeue --user=<username>.

To cancel all jobs you have submitted that are either running or waiting in the queue, enter the following:

$ scancel --user=<username>

Job Limits

To distribute computational resources fairly to all Midway users, the RCC sets limits on the amount of computing resources that may be requested by a single user at any given time.

The maximum run-time for an individual job is 36 hours. This applies to all batch and interactive jobs submitted to nodes in the general-access partitions (sandyb, westmere, bigmem, broadwl, and so on). Groups participating in the cluster parternership program may customize resources limits for their partitions.

Additional information on limits, such as the maximum number of CPUs that can be requested by a user at any one time, or the number of jobs that can be submitted concurrently on a given partition, can be found by entering the command accounts qos on any login or compute node on midway. Observe that these limits are often different depending on the partition.

Usage limits may change, so accounts qos will always give the most up-to-date information.

If your research requires a temporary exception to a particular limit, you may apply for a special allocation. Special allocations are evaluated on an individual basis and may or may not be granted.