Search Results for “security policy” – Page 3

Search Results for: security policy

Frequently Asked Questions (FAQ)

LOGIN AND AUTHENTICATION

My login is slow or my batch commands are slow

Nine times out of ten, slowness at login, starting file transfers, failed SFTP sessions, or slow batch command starts is caused by un-needed module loads in your .bashrc

We do not recommend putting multiple module loads in your .bashrc as each and every new shell you or your jobs create will call those module loads. It is recommended that you put your module loads in your job scripts so that you are not loading un-needed modules and waiting on those module calls to complete before commencing the job. Alternately, you can create a login script or alias containing your frequently used modules that you can run when you need to use them.

Either way, try to keep any module loads in your .bashrc down to a bare minimum, calling only those modules that you absolutely need in each and every login or job.

Additionally, as time goes on modules change or are removed. Please ensure you remove any deprecated modules from your .bashrc or other scripts. For example, the legacy modules no longer exist. So if you have a call to module load legacy and any of the legacy modules, or if you have source new-modules.sh your login will be delayed as the module system searches for and then times out on those non-existent modules.

My alternate shell (csh, tcsh, etc.) doesn’t work right

Having a non-standard default shell will cause problems and does not allow us to set global environmental defaults for everyone. As 2019 we will no longer change the default shell on any account or support the use of alternate shells as default login shell.

Users can, of course, still launch an alternate shell once logged in. Built-in shells such as sh, zsh, and csh already exist on most nodes.

SSH key error, DNS spoofing message

If you are getting SSH key or host errors, see this page.

SFTP exits after a few seconds

When connecting via a SFTP client like Filezilla, if you experience a short delay and then disconnection, this is most likely an issue caused by your .bashrc

During SFTP connections, your .bashrc will be evaluated just as if you were logging in via SSH. If you’ve added anything to your .bashrc that attempts to echo to the terminal/standard out, this will cause your SFTP client to hang and then disconnect.

You can either remove the statement in your .bashrc that is writing output (an echo statement, a call to an app or module that sends a message to standard out, etc.) -or- you can put the offending statement into an evaluation clause that first checks to see if this is a interactive login, like so:

if [ “$SSH_TTY” ]
then
echo “SFTP connections won’t evaluate the things inside this clause."
echo "Only real login sessions will.”
fi

What happens to my account when I leave/graduate?

Please see this page: What happens to my FASRC account when I leave Harvard?

How do I request membership in additional lab groups?

Please see Additional Group Membership

Can I use SSH keys to log in without a password?

No. Our cluster login relies on two-factor authentication. This makes using key-based authentication impractical.

How do I get a Research Computing account?

Before You Sign Up

If you are unsure whether you qualify for an RC account, please see Qualifications and Affiliations. More information on using the signup tool can be found here.

Please Note: You may have only one RC account. If you need to add cluster access or membership in a different/additional lab group, please submit a help ticket. Please do not sign up for a second account. This is unnecessary and against our account policies.

The Process

To request an account to access resources operated by Research Computing. (Cluster, Storage, Software Downloads, Workstation access, Instrument sign-up, etc.), please proceed to the

Account Request Tool

PLEASE NOTE: Do not select FACULTY as your job type is you do not have a faculty appointment. If you are a researcher with additional rights (fellowship, PI-like rights, funding, etc.), please select STAFF or POSTDOC. Faculty accounts are intended only for those holding an active Associate Professor or higher appointment.

Once you’ve submitted the request, the process is:

If You Selected: Internal/Using Harvard Key to verify your information and qualifications:

The request is on hold while the PI is asked to approve or reject it.
Once approved, the account is finalized and set up.
Once finalized, you receive an automated email confirmation with your new account information and instructions for setting the password.

If You Selected: External/Not using Harvard Key to verify your information and qualifications:

The request goes to RC personnel to check that it is complete and meets affiliation requirements.
Once approved by RC, an email is sent to your PI to approve/reject the request.
The request is on hold while the PI is asked to approve or reject it.
Once approved, we finalize the account on our side (during business hours).
Once finalized, you receive an automated email confirmation with your new account information and instructions for setting the password..

You can then proceed to set up your OpenAuth token and get connected to the cluster. The turnaround time is directly related to the PI/Sponsor’s approval of the account. External accounts are reviewed by RC staff during business hours and generally vetted and sent on to the PI/Sponsor for approval within one business day

NOTE! If you request “Cluster Use” (the ability to run jobs on the cluster), attend one of our monthly New User Trainings or watch our Introduction videos.

Can someone else approve my account request?

Initially, only the PI for a lab can sponsor and approve new accounts under their lab group. They may also at any point designate another account holder(s), such as a lab admin or faculty assistant, in their lab as additional approvers by contacting FASRC directly. Approval to add additional approvers can only come directly from the PI to FASRC (e.g. – a forwarded email is not sufficient, the PI needs to contact us directly.)

Can I share an account? – Account Security Policies

The sharing of passwords or login credentials is not allowed under RC and Harvard information security policies. Please bear in mind that this policy also protects the end-user.

Sharing credentials removes the ability to audit and accountability for the account holder in case of account misuse. Accounts which are in violation of this policy may be disabled or otherwise limited. Accounts knowingly skirting this policy may be banned.

If you find that you need to share resources among multiple individuals, Faculty can approve accounts for outside collaborators to their lab groups. Otherwise, please contact us and we will be happy to assist you with finding a safe and secure way to do so.

How do I login to the FASRC cluster?

See our Access and Login page.
and/or
Our terminal access page.

How do I reset my Research Computing account password?

Please click here to reset your Research Computing account password using your email address.

This will send an email to you with a one-time use link to set a new password.

Please note: Your username is not your email address. Your email address is used here only for password resets and to contact you.

How do I unlock my locked Research Computing account?

Typically, after entering the incorrect password multiple times your account will become locked. Once your account is locked, your account will automatically unlock after ~ 5 – 10 minutes. If your account remains locked for longer please contact us.

How do I install and launch OpenAuth?

If you do not yet have an account, see: How do I get a Research Computing account? For additional instructions, see: Account Signup

Setting Up Your OpenAuth Token

Visit https://two-factor.rc.fas.harvard.edu/ to start setup of OpenAuth.
A login box will appear. Log in with your FAS RC username and password (your username is not your email address or Harvard Key, it is the short username you initially set up when requesting an account. Example: jsmith )
After logging in, allow a few seconds as the site generates your token.
A page will be displayed outlining next steps
Await an email. This email will contain a link to your personalized token. You can download the Java applet or use the QR code on that page to add your RC token in Google Authenticator or Duo Mobile

Since the site uses email verification to authenticate you, you must also have a valid account and email address on record with Research Computing. All OpenAuth tokens are software-based, and you will choose whether to use a smart phone or java desktop app to generate your verification codes. Java 1.6 or higher is required for the desktop app.

You will need to use OpenAuth when accessing the Research Computing VPN and logging into the FAS RC cluster.

How do I logon to the Research Computing VPN?

Please see our VPN setup guide here.

Linux users please see our guide to using OpenVPN here.

I need an AWS account and/or Amazon AWS virtual machine

AWS offerings are through HUIT. Please see https://cloud.huit.harvard.edu/ or contact ithelp@harvard.edu

FILESYSTEMS AND AUTHORIZATION

Where is ftp?

Modern secure transfer protocols like SFTP and SCP secure data during transit and should be used when moving files from one place to another. However you may still need to use plain, un-secured FTP to download data sets or other files from remote locations while logged into the cluster.

While we do not offer the largely outmoded ‘ftp’ program on the cluster, we do offer the feature-rich and largely command compatible ‘lftp’. From any login or compute node type ‘man lftp’ to see its usage and options.

How do I request membership in additional lab groups?

Please see Additional Group Membership

What’s the best way to transfer my data?

INTERNAL
See our ‘Transferring data on the cluster‘ page for a list of options and best practices for data transfer within the cluster.

EXTERNAL
For transferring data to and from the cluster, see ‘Transferring data externally‘.

How do I access my cluster home directory from my laptop?

FASRC cluster home directories are available through SAMBA and so can be mounted as a network drive on Mac, Windows, and Linux computers. See the Mounting Storage page for specific instructions on how to mount the directory.

How do I check how much space I’ve used, what’s my quota?

See Checking quota and usage for information on how to use the FASRC quota tool to check quota and storage usage. FASRC filesystems supported by this tool are described in Data Storage Workflow.

I accidentally deleted my data, how do I get it back?

Your home directory has periodic snapshots taken. These snapshots are of your home directory files from various recent points in time. They are in a hidden directory named .snapshot, within every other directory in your home directory. The command ls -a will not show these, but you can ls .snapshot directly, and cd .snapshot to go into the directory.

In the .snapshot folder you will see “hourly” “daily” “monthly” folders with the date of the snapshots. Traverse (cd) to the snapshot folder corresponding to the period you wish to restore data from. From there you can simply copy the relevant files back into your home folder using your favorite file copy tool (rsync, cp, etc.)

Lab directory backups are for system-wide disaster recovery only, they are handled separately and do not have snapshot capabilities, they are not intended to recover accidental file deletions. Please contact FASRC if you have any questions.

Please also see our Storage document for more info.

Why are all my files executable?

You may notice that the x (execute) bit is set on all your files:

[username@boslogin01 ~]# ls -l myfile.txt-rwxr–r– 1 username groupname 3029 Aug 20 03:10 myfile.txt

Furthermore, chmod does not remove it:

[username@boslogin01 ~]# chmod u-x myfile.txt

[username@boslogin01 ~]# ls -l myfile.txt

-rwxr–r– 1 username groupname 3029 Aug 20 03:10 myfile.txt

This is a feature, a result of the storage system doing mixed Unix-style and Windows-style permissions. If this is causing a problem for you, please contact FASRC.

Why does my `UMASK` not work?

You may also notice that your UMASK environment variable does not work as expected:

[username@boslogin01 ~]# umask 002

[username@boslogin01 ~]# touch newfile.txt

[username@boslogin01 ~]# ls -l newfile.txt

-rwx—— 1 username groupname 3029 Aug 20 03:10 newfile.txt

Normally, the outcome would be -rw-rw-r--. If this is causing a problem for you, please contact FASRC.

Is my home directory available as a network filesystem share?

Yes, your cluster home directory is available as a network filesystem share to which you can directly connect your own desktop or laptop. The technical protocol for this is called CIFS or Samba, so you will often hear us refer to it in that way. On Windows, this is also referred to as mapping a network drive, and on a Mac it is called connecting to a server.

In all cases, you need your RC username, password, server name, and path. Please see the Mounting Storage document for detailed information.

I am seeing weird errors about file locking with HDF5. What do I do?

VAST filesystems (netscratch and holylabs) have known issues with file locking for HDF5 and other things that do sophisticated parallel IO. It is recommended that you build and use the vast-preload-lib for your specific MPI version. You may also need to set export HDF5_USE_FILE_LOCKING=FALSE

SOFTWARE

I need cluster access to Gaussian

Please contact us if you require Gaussian access. It is controlled on a case-by-case basis and requires membership in a security group.

To see all available versions of Gaussian, visit the All Modules page and Search for ‘gaussian’.

I need to download GaussView or MOE

FASRC users can download these clients from our Downloads page. You must be connected to the FASRC VPN to access this page. Your FASRC username and password are required to log in.
FASRC no longer has access to a JMP Pro/Genomics license. Please see the JMP site for licensing details. FASRC does provide SAS 9.4 for use in jobs on the cluster.

I need to download Geneious Pro or MOE (only available for FAS users)

FAS members can download these clients from our Downloads page. You must be connected to the FASRC VPN to access this page. Your FASRC username and password are required to log in. Not for use by members of other schools or external users.

Geneious should work from any wired Harvard Science department network or when connected to either the FAS or RC VPN. A VPN connection will be required if you are using HARVARD WIRELESS network connection or a network connection not allocated to departments within Harvard’s FAS Division of Science. For details on using the RC VPN (@fasrc), please see FASRC VPN setup.

I can’t search for R

Unfortunately, having a single letter as the name of an application makes searching problematic.

Here are links to our R Basics and R Packages pages

Where is FTP?

How do I load a module or software on FASRC cluster?

Step 1: Login to the cluster through your Terminal window. Please see here for login instructions.

Step 2: Load a module/software by typing: module load MODULENAME. Replace MODULENAME with the specific software you want to use. A complete listing of modules can be found on the module list page.

To see what modules you have loaded type: module list

To unload a module type: module unload MODULENAME

Details can be found in the modules section of the Running Jobs page.

FileZilla: I have to enter my OpenAuth code every 30 seconds

If you are using Filezilla to transfer files to the cluster, and you are prompted frequently (like every 30 seconds!) to enter your username and/or OpenAuth token code, then most likely you did not configure FileZilla according to our instructions. You must limit the number of connections to 1, else Filezilla will spawn more connections, each requiring you to authenticate.

Please see this document on how to set the connection limit and avoid the OpenAuth challenge frustration while transferring files to and from the cluster.

Git/Github: 403 Forbidden while accessing https://github.com…

If you issue a git push to a cloned repository, you might receive the following error:

error: The requested URL returned error: 403 Forbidden while accessing https://github.com/yourusername/planets.git/info/refs fatal: HTTP request failed

Authorization to Github repositories on the cluster is can be a little tricky. Please follow our instructions at or git and github on the FASRC cluster.

How do I run a Matlab script on the FASRC cluster?

To run a Matlab script (with no graphical interface component) on the cluster, login using your preferred terminal application then activate the application by loading the module.

module load matlab/R2018b-fasrc01

Then, assuming your script is named calc.m, either run it through an interactive session

salloc --mem 1000 -p test matlab -nojvm -nodisplay -nosplash < calc.m

or use the matlab command in a batch script

#!/bin/bash
#SBATCH -o calc.out 
#SBATCH -o calc.err 
#SBATCH -p serial_requeue 
#SBATCH -n 1 
#SBATCH --mem 1000 
#SBATCH -t 1000

matlab -nojvm -nodisplay -nosplash < calc.m

Make sure that `calc.m` finishes with an `exit` command. Otherwise, the process will hang waiting for further input.

Perl modules: Can’t locate XX.pm in @INC

Perl modules have been developed over the past 15 to 20 years, and the installation method has changed significantly. Unfortunately, you might run into a program that needs to install a really old Perl module, and its installation is just not behaving properly under the new installation methods. You might see something like the following:

[bfreeman@holylogin01 PfamScan]$ ./pfam_scan.pl --help Can't locate Data/Printer.pm in @INC (@INC contains: /n/sw/fasrcsw/apps/Core/perl-modules.....

The remedy can be rather simple:
1. Follow our new lmod – Perl instructions here on setting up your home directory for installing Perl modules ‘locally’.

Note that the export PERL5LIB command must include both $LOCALPERL and $LOCALPERL/lib/perl5 (it’s subdirectory) as some installation routines honor one; some the other.

2. Sometimes, you might need to install the module manually. Try both the Makefile.PL build and the Build.PL build if one or the other doesn’t work.

3. In CPAN, you can do this manual install method without the hassle of the download process:

cpan look Data::Printer

This latter command will download the module and unpack it for you, and leave you at the shell, where you can try either the Makefile.PL or Build.PL build process.

Illegal Instruction

If you are getting an error indicating an illegal instruction that likely means that your code was built on a different processor type than the one you are running on. The cluster has a variety of different hardware and if your code tries to leverage instructions specific to that hardware then the code cannot run on other types of hardware. To resolve this error you will either need to build your code with out the hardware specific instruction sets, or tell the scheduler via the --constraint option to only run your jobs on the specific hardware types you have built your code for. A full list of constraints can be found on the Running Jobs page.

Installing LaTeX packages

The TeX Live distribution that is installed on FASRC cluster nodes includes a core set of LaTeX packages. Missing packages needed for a specific LaTeX document may be installed in your home directory. One-time setup is needed beforehand:

tlmgr init-usertree
tlmgr --usermode option repository https://www.texlive.info/tlnet-archive/2018/03/03/tlnet/

Then, e.g., given the following error when using LaTeX:

! LaTeX Error: File `ucharcat.sty' not found.

The missing package may be installed in your home directory as follows:

tlmgr --ignore-warning --usermode install ucharcat

JOBS AND SLURM

How do I know what partitions I have access to?

The spart command can be used find a quick summary of this information. scontrol show partition and sinfo will also give more detailed information about the various partitions you have rights to use.

How do I know what memory limit to put on my job?

Add to your job submission:

#SBATCH --mem X

where X is the maximum amount of memory your job will use per node, in MB. The larger your working data set, the larger this needs to be, but the smaller the number the easier it is for the scheduler to find a place to run your job. To determine an appropriate value, start relatively large (job slots on average have about 4000 MB per core, but that’s much larger than needed for most jobs) and then use sacct to look at how much your job is actually using or used:

sacct -o MaxRSS -j JOBID

where JOBID is the one you’re interested in. The number is in KB, so divide by 1024 to get a rough idea of what to use with –mem (set it to something a little larger than that, since you’re defining a hard upper limit).

For more information see here.

How do I figure out how efficient my job is?

The jobstats command can be used to report job efficiency and help determine CPU/GPU/memory/time to allocate to future jobs.

See Job Efficiency and Optimization Best Practices for tips on how to right-size job allocations and optimize use of allocated resources.

Will single core/thread jobs run faster on the cluster?

The cluster cores, in general, will not be any faster than the ones in your workstation, in fact they may be slower if your workstation is relatively new. While we have a variety of chipsets available on the cluster, most of the cores are AMD and will be slower than many Intel chips, which are most common in modern desktops and laptops. The reason we use so many AMD chips is that we could purchase a larger number of cores and RAM this way. This is the power of the cluster. The cluster isn’t designed to run a single core code as fast as possible as the chips to do that are expensive. Rather you trade off raw chip speed for core count. Then you gain speed and efficiency via parallelism. So the cluster excels at multicore jobs (using threads or MPI ranks) or doing many jobs that take a single core (such as parameter sweeps or image process). This way you leverage the parallel nature of the cluster and the 60,000 cores available.

So if you have a single job, the cluster isn’t really a gain. If you have lots of jobs you need to get done, or your job is too large to fit on a single machine (due to RAM or its parallel nature), the cluster is the place to go. The cluster can also be useful for offloading work from your workstation. That way you can use your workstation cores for other tasks and offload the longer running work onto the cluster.

In addition since the cluster cores are a different architecture from your workstation one needs to be aware that the code will need to be optimized differently. This is where compiler choice and compiler flags can come in handy. That way you can get the most out of both sets of cores. Even there you may not get the same performance out of the cluster as your local machine. The main processor we have on the cluster is now 4 years old, and if you are using serial_requeue you could end up on hardware bought today to stuff purchased 7 years ago. There is about a factor of 2-4 in performance in just the natural development of processor technology.

My login is slow or my batch commands are slow

Nine times out of ten, slowness at login, starting file transfers, failed SFTP sessions, or slow batch command starts is caused by un-needed module loads in your .bashrc

Either way, try to keep any module loads in your .bashrc down to a bare minimum, calling only those modules that you absolutely need in each and every login or job.

How do I request membership in additional lab groups?

Please see Additional Group Membership

Can I query SLURM programmatically?

I’m writing code to keep an eye on my jobs. How can I query SLURM programmatically?

We highly recommend that people writing meta-schedulers or that wish to interrogate SLURM in scripts do so using the squeue and sacct commands. We strongly recommend that your code performs these queries once every 60 seconds or longer. Using these commands contacts the master controller directly, the same process responsible for scheduling all work on the cluster. Polling more frequently, especially across all users on the cluster, will slow down response times and may bring scheduling to a crawl. Please don’t.

SLURM also has an API that is documented on the website of our developer partners SchedMD.com.

Are their policies or guidelines for using the cluster responsibly?

Yes. Please see out Customs and Responsibilities page.

How do I submit a batch job to the FASRC cluster queue with SLURM?

Step 1: Login to cluster through your Terminal window. Please see the Access and Login page for login instructions.

Step 2: Run a batch job by typing: sbatch RUNSCRIPT. Replace RUNSCRIPT with the batch script (a text file) you will use to run your code.

The batch script should contain #SBATCH comments that tell SLURM how to run the job.

#!/bin/bash
#SBATCH -n 1 #Number of cores 
#SBATCH -t 5 #Runtime in minutes
#SBATCH -p serial_requeue #Partition to submit to 
#SBATCH --mem-per-cpu=100 #Memory per cpu in MB (see also --mem) 
#SBATCH -o hostname.out #File to which standard out will be written 
#SBATCH -e hostname.err #File to which standard err will be written

See the batch submission section of the Running Jobs page for detailed instructions and sample batch submission scripts.

Note: You must declare how much memory and how many cores you are using for your job. By default SLURM assumes you need 100 MB. The script assumes that it is running in the current directory and will load your .bashrc.

How do I submit an interactive job on the cluster?

Step 1: Log in to the cluster through your Terminal window. Please see here for login instructions.

Step 2: Run an interactive job by typing: salloc -p test MYPROGRAM

This will open up an interactive run for you to use. If you want a bash prompt, type: salloc --mem 500 -p test

If you need X11 forwarding type: salloc --mem 500 -p test --x11 MYPROGRAM

This will initiate an X11 tunnel to the first node on your list.

See also the interactive jobs section of the Running Jobs page.

How do I view or monitor a submitted job?

Step 1: Login to the cluster through your Terminal window. Please see the Access and Login page for login instructions.

Step 2: From the command line type one of three options: smap, squeue, or showq-slurm

If you want more details about your job, from the command line type: sacct -j JOBID

You can view the runtime and memory usage for a past job by typing: sacct -j JOBID --format=JobID,JobName,MaxRSS,Elapsed, where JobID is the numeric job ID of a past job.

See the Running Jobs page for more details on job monitoring.

My job is PENDING. How can I fix this?

How soon a job is scheduled is due to a combination of factors: the time requested, the resources requested (e.g. RAM, # of cores, etc), the partition, and one’s FairShare score.

Quick solution? The Reason column in the squeue output can give you a clue:

If there is no reason, the scheduler hasn’t attended to your submission yet.
Resources means your job is waiting for an appropriate compute node to open.
Priority indicates your priority is lower relative to others being scheduled.

There are other Reason codes; see the SLURM squeue documentation for full details.

Your priority is partially based on your FairShare score and determines how quickly your job is scheduled relative to others on the cluster. To see your FairShare score, enter the command sshare -u RCUSERNAME. Your effective score is the value in the last column, and, as a rule of thumb, can be assessed as lower priority ≤ 0.5 ≤ higher priority.

In addition, you can see the status of a given partition and your position relative to other pending jobs in it by entering the command showq-slurm -p PARTITION -o. This will order the pending queue by priority, where jobs listed at the top are next to be scheduled.

For both Resources and Priority squeue Reason output codes, consider shortening the runtime or reducing the requested resources to increase the likelihood that your job will start sooner.

Please see this document for more information and this presentation for a number of troubleshooting steps.

SLURM Errors: Job Submission Limit (per user)

If you attempt to schedule more than 10,000 jobs (all inclusive, both running and pending) you will receive an error like the following:

sbatch: error: Batch job submission failed: Job violates accounting/QOS policy (job submit limit, user’s size and/or time limits)

For more info about being a good cluster neighbor, see: https://docs.rc.fas.harvard.edu/kb/responsibilities/

SLURM Errors: Device or resource busy

What’s up? My SLURM output file terminates early with the following error:

"slurmstepd: error: _slurm_cgroup_destroy: problem deleting step cgroup path /cgroup/freezer/slurm/uid_57915/job_25009017/step_batch: Device or resource busy"

Well, usually this is a problem in which your job is trying to write to a network storage device that is busy — probably overloaded by someone doing high amounts of I/O (input/output) where they shouldn’t, usually on low throughput storage like home directories or lab disk shares.

Please contact RCHelp about this problem, giving us the jobID, the filesystem you are working on, and additional details that may be relevant. We’ll use this info to track down the problem (and, perhaps, the problem user(s)).

(If you know who it is, tap them on the shoulder and show them our Cluster Storage page.)

SLURM errors: Job cancelled due to preemption

If you’ve submitted a job to the serial_requeue partition, it is more than likely that your job will be scheduled on a purchased node that is idle. If the node owner submits jobs, SLURM will kill your job and automatically requeue it. This message will appear in your STDOUT or STDERR files you indicated with the -o or -e options. This is simply an informative message from SLURM.

SLURM Errors: Memory limit

Job <jobid> exceeded <mem> memory limit, being killed:

Your job is attempting to use more memory than you’ve requested for it. Either increase the amount of memory requested by --mem or --mem-per-cpuor, if possible, reduce the amount your application is trying to use. For example, many Java programs set heap space using the -Xmx JVM option. This could potentially be reduced.

For jobs that require truly large amounts of memory (>256 Gb), you may need to use thebigmem SLURM partition. Genome and transcript assembly tools are commonly in this camp.

See this FAQ on determining how much memory your completed batch job used under SLURM.

SLURM Errors: Node Failure

JOB <jobid> CANCELLED AT <time> DUE TO NODE FAILURE:

This message may arise for a variety of reasons, but it indicates that the host on which your job was running can no longer be contacted by SLURM. Not a good sign. Contact RCHelp to help with this problem.

SLURM errors: Socket timed out. What?

If the SLURM master (the process that listens for SLURM requests) is busy, you might receive the following error:

[bfreeman@holylogin02 ~]$ squeue -u bfreeman squeue: error: slurm_receive_msg: Socket timed out on send/recv operation slurm_load_jobs error: Socket timed out on send/recv operation

Since SLURM is scheduling 1 job every second (let alone doing the calculations to schedule this job on 1 of approximately 100,000 compute nodes), it’s going to be a bit busy at times. Don’t worry. Get up, stretch, pet your cat, grab a cup of coffee, and try again.

SLURM Errors: Time limit

JOB <jobid> CANCELLED AT <time> DUE TO TIME LIMIT:
(or you may also see ‘Job step aborted’ when using salloc/srun)

Either you did not specify enough time in your batch submission script, or you didn’t specify the amount of time and SLURM assigned the default time of 10 minutes. The -t option sets time in minutes or can also take D-HH:MM form (0-12:30for 12.5 hours). Submit your job again with a longer time window.

What is Fair-Share?

FairShare is a score that determines what priority you have in the scheduling queue for your jobs. The more jobs you run, the lower your score becomes, temporarily. A number of factors are used to determine this score — please read this Fairshare document for more information.

To find out what your score is, enter `sshare -U` in your terminal session on the cluster to see a listing for your group (this is not your individual score, but an aggregate for your group). In general, a score of 0.5 or above means you have higher priority for scheduling.

Example of a fairly full Fairshare:

$ sshare -U
Account User RawShares NormShares RawUsage EffectvUsage FairShare
------------ ----- ------ -------- ------- ------------- ----------
jharvard2_lab jharv parent 0.000936 171281 0.000003 0.997620

Example of a depleted Fairshare:

$ sshare -U
Account User RawShares NormShares RawUsage EffectvUsage FairShare
------------ ----- ------ -------- ------- ------------- ----------
jharvard_lab johnh parent 0.000936 361920733 0.007145 0.005046

See also: Managing FairShare for Multiple Groups if you belong to more than one lab group

For further information, see the RC fairshare document.

Can I send mail from the cluster?

The short answer is no. You can receive job emails as covered in our Running Jobs doc, but you cannot send emails from cluster nodes.

The longer answer is that the FASRC cluster could easily be weaponized to send bulk email if we allowed this and could cause a portion of Harvard’s IP range (or even all of Harvard’s IP range) to be added to a deny list. The cluster is intended as a research compute platform, and its nodes, while running Linux, are not the same as workstation or server nodes one might be used to. Any post-processing or use of such tools as email or printing should be done using another system.

I see dummy4XD jobs, but I didn’t submit them?

We use a tool called XDMoD for record keeping. In order to ensure our usage statistics are correct, dummy jobs are submitted on behalf of users. You do not need to delete them; they run very quickly and your fairshare is not used for them. It is safe to ignore these jobs.

I see nodes marked as DRAINING, DOWN, or COMPLETING in the partition that I am using what can I do?

When you see nodes in this state there is nothing you need to do and there is no need to notify FASRC staff. At any given time there will be a number of nodes that are in a state of DRAINING, DOWN, or seem to be stuck in the COMPLETING state. This generally means that the scheduler has identified one or more problems with these nodes and has set these states so that the nodes will not accept any jobs until the problem is resolved. FASRC staff patrol the cluster for broken nodes and will open the nodes once they are fixed. If you notice a node is still closed then that just means that the FASRC staff have deemed it not ready for service yet. A reason for the node closure is noted in slurm which you can see by doing scontrol show node NODENAME. If you are curious what these reasons mean or if you see INCXXXXXX (which indicates a hardware issue we are dealing with) you can contact us to find out more details.

BILLING

Data Storage Billing

VDI (Open OnDemand)

Why is my Jupyter notebook VDI session terminated right after it starts?

This problem is common when there is a conda initialize section in your .bashrc file located in your home directory (more about .bashrc). The conda initialize section was added when, at some point, you used the command conda init. We strongly discourage the use of conda init. Instead use source activate environment_name, for more details, refer to our Python (Anaconda) page.

To solve this problem, delete or comment out the conda initialize section of your .bashrc and create a new Jupyter notebook VDI session.

Home and Lab directories

Please see the Data Storage on our main website information on other storage options and for clarification on any unfamiliar terms.

This page describes the resources which are available to each user account and lab, and is a guide for day-to-day usage.

Home Directories

Every user whose account has cluster access receives a 100 GB home directory. Your initial working directory upon login is your home directory. This location is for your use in storing everyday data for analysis, scripts, documentation, etc. This is also where files such as you .bashrc reside. Home directories paths look like /n/homeNN/XXXX where homeNN is home01–home15 and XXXX is your login. For example, user jharvard’s home directory might be /n/home12/jharvard. You can also reach your home directory using the Unix shortcut ~, as in: cd ~

Size Limit: 100GB (hard limit)
Availability: All cluster nodes. Can be mounted on desktops and laptops
Backup: Daily snapshots. Retained for 2 weeks
Retention policy: indefinite
Performance: Moderate. Not appropriate for I/O intensive or large numbers of jobs
Cost: Provided with each user account

Your home volume has good performance for most simple tasks. However, I/O intensive or large numbers of jobs should not be processed in home directories. Widespread computation against home directories would result in poor performance for all users. For these types of tasks, the scratch filesystem is better suited.

Home directories are private to your account and will follow you no matter should you change labs, but are not suitable for storing HRCI/level 3 or above data. This is a violation of Harvard security policies. Home directories are tied to the user account, not the sponsoring lab or PI, and are governed by the Harvard Policy on Access to Electronic Information. Home directories follow the account throughout its life-cycle.

Your home directory is exported from the disk arrays using CIFS/SMB file protocols and so can be mounted as a ‘shared drive’ on your desktop or laptop. Please see this help document for step-by-step instructions.

Home directories are backed up into a directory called .snapshot in your home. This directory will not appear in directory listings. You can cd or ls this directory specifically to make it visible. Contained herein are copies of your home directory in date specific subdirectories. Hourly, daily, weekly snapshots can be found. To restore older files, simply copy them from the correct .snapshot subdirectory. NOTE: If you delete your entire home directory, you will also delete the snapshots. This is not recoverable.

The 100 GB quota is enforced with a combination of a soft quota warning at 95GB and a hard quota stop at 100 GB. Hitting quota during processing of large data sets can result in file write/read failures or segmentation faults. You can check your usage using the df command: df -h ~ (where ~ is the unix shortcut for ‘home’)

TIP: If you are trying to determine usage, you might try using du -h -d 1 ~ to see the usage by sub-directory, or du -ax . | sort -n -r | head -n 20 to get a sorted list of the top 20 largest.

When attempting to log in when your home directory is over quota, you will often see an error in the .Xauthority file:
/usr/bin/xauth: error in locking authority file .Xauthority Logging into an NX or other virtual service will fail as the service cannot write to your home directory.

When at or over quota, you will need to remove unneeded files. Home directory quotas are global and cannot be increased for individuals. You may be able to use lab or scratch space to assist with copying or moving files from your home directory to free up space.

Lab Directories

Each lab that uses the cluster receives a 4 TiB lab directory (as of 2025 – these will reside in /n/holylabs). This location is for each lab group’s use in storing everyday data for analysis, scripts, documentation, etc. Each such lab will have a directory on our high-performance scratch filesystem (see below).

Size Limit: 4TiB (hard limit), 1 million files
Availability: All cluster nodes. Cannot be mounted on desktops and laptops
Backup: Highly redundant, no backups
Retention policy: Duration of the lab group
Performance: Moderate. Not appropriate for I/O intensive or large numbers of jobs
Cost: Provided with each lab group

Lab directories have good performance for most simple tasks. However, I/O intensive or large numbers of jobs should not be processed in lab directories. Widespread computation against lab directories would result in poor performance for all users. For these types of tasks, the scratch filesystem is better suited.

This lab directory is owned by the lab’s PI and is intended only to be used for research data on the cluster. research storage should not be used for administrative files and data.

Lab directories are not suitable for storing HRCI/level 3 or above data. This is a violation of Harvard security policies.

The 4 TB quota is enforced with a combination of a soft quota warning and a hard quota stop at 4 TB. Hitting quota during processing of large data sets can result in file write/read failures or segmentation faults. If your lab requires additional storage, see our Data Storage page for a list of available storage options.

> Running Jobs

Introduction

Faculty of Arts and Sciences Research Computing (FASRC) hosts several collections of computers in what are called clusters. Each cluster is large number of individual compute servers networked together with a high speed interconnect and integrated with storage (see our Data Management guide for more). To manage work on these clusters FASRC uses Slurm.

Slurm is a open source scheduler from SchedMD. The job of Slurm is:

1. To govern what user gets what resources on the cluster and when.
2. To create allocations for individual units of work which are called jobs.
3. To ensure maximum utilization of the cluster.
4. To keep a historical record of usage.

Users interact with Slurm by submitting a job to the scheduler. The scheduler then puts that job in the pending queue for the selected subsection of the cluster (called a partition) for consideration. The scheduler will weight the job’s priority based on the users prior usage to ensure a fair distribution of resources. It will then try to schedule the highest priority work by playing a large scale game of Tetris. In addition Slurm will take lower priority jobs and try to fit them into various gaps it finds in order to maximize usage without impacting the time when the higher priority work would run.

Below we will walk you through how to submit jobs to the scheduler for work. We will also discuss how the cluster is organized and some best practices for use. For more details on the architecture of the cluster, please see our Job Efficiency and Optimization Best Practices page.

Getting Started

To submit jobs you will first need to set up your account. Once you’ve gone through the account setup procedure, you can login to the cluster via ssh to a login node and/or use Open OnDemand. The guide below assumes that you will be using the command line (CLI) for interaction with Slurm.

FASRC cluster nodes run the Rocky distribution of the Linux operating system and commands are run under the bash shell. There are a number of Linux and bash references, cheat sheets and tutorials available on the web. FASRC’s own training is also available.

Storage and Scratch on the Cluster

Cluster nodes have file systems mounted for use by labs and individuals to store both on a temporary (called scratch) and long term basis. The Data Storage page covers the various storage options. Please use the appropriate storage for your jobs as each storage type has different purposes and performance characteristics.

Slurm Documentation

Comprehensive documentation for Slurm can be found at the official Slurm website. Note that these docs are always for the latest version of Slurm, while FASRC tries to keep up with the latest version you will want to cross check the version we run against the version the docs are for. To find the version of Slurm the cluster is running do sinfo --version.

You can also get documentation on individual commands by using the unix man command. This command will show you the manual for the command for the version of Slurm the cluster is using. For instance if you want the manual for the sinfo command you would run: man sinfo

Some other useful documentation sites are:

Summary of Slurm Commands

The table below shows a brief list of common Slurm commands. These commands are described in more detail below along with links to the Slurm doc site.

What you want to do	SLURM	SLURM Example
Submit a batch serial job	sbatch	`sbatch runscript.sh`
Run a script or application interactively (do not use salloc on FASSE)	salloc	`salloc -p test -t 10 --mem 1G [script or app]`
Start interactive session (do not use salloc on FASSE)	salloc	`salloc -p test -t 10 --mem 1G`
Kill a job	scancel	`scancel JOBID`
View status of your jobs	sacct	`sacct -u USERNAME`
Check job by id number	sacct	`sacct -j JOBID`
Check efficiency of job	jobstats	`jobstats JOBID`
List of available partitions	spart	`spart`
Check current partition queue state	showq	`showq -o -p PARTITIONNAME`
Details on current job, node, partition	scontrol	`scontrol show job JOBID` `scontrol show node NODENAME` `scontrol show partition PARTITIONNAME`
Schedule recurring batch job	scrontab	see scrontab document for example
Check fairshare	sshare	`sshare -U`

Slurm Global Limits and Defaults

Before submitting any jobs users should familiarize themselves with:

FASRC has set several global limits that users should be aware of and should plan around. These limits exist to prevent any one person from taking over the cluster and also serve to prevent the cluster being overwhelmed due to poorly formed jobs. Users must work within these limits and should plan their work accordingly. This is typically done by breaking up their workflow into smaller chunks or by deliberately serializing their jobs to increase the job time and decrease the number of cores needed. The limits are as follows:

Maximum Number of Jobs per User: 10,100. This is meant to prevent any one user from monopolizing the cluster.
Maximum Array Size: 10,000. This is both array index and size. This is meant to prevent any one user from monopolizing the cluster. Note that each array index counts as a single job for purposes of the Maximum Number of Jobs per User, so this is intentionally redundant.
Maximum Number of Steps: 40,000. A job step is recorded by slurm for each invocation of srun by a job. This is meant to prevent run-away jobs.

All other limits are partition or node dependent. More on that below.

FASRC also sets the following defaults if nothing is requested:

Core Count: 1
Memory: 100 MB
GPU Count: 0
Partition: serial_requeue
Time: There is no default time set. Users must always declare time.

Users can set their own defaults by setting a definition file in $HOME/.slurm/defaults, for more see the CLI Filter doc.

Slurm Partitions

Partitions are a block of nodes on the cluster with their own scheduling policy. Partitions have various limits governing what types of jobs are appropriate to run in them. When a job is submitted it schedules to the specified partition(s) and then joins the pending queue. When the job is scheduled in a partition it will join the running queue for that partition. You can find out what partitions you have access to using the spart command. To learn more about a given partition run: scontrol show partition PARTITIONAME. To learn more about an individual node run: scontrol show node NODENAME. Below is a list of the public partitions on Cannon (FASSE can be found here).


Partition	Nodes	Cores per Node	CPU Core Types	Mem per Node (GB)	Time Limit	Max Jobs	Max Cores	GPU Capable?	/scratch size (GB)
sapphire	184	112	Intel “Sapphire Rapids”	990	3 days	none	none	No	396
shared	310	48	Intel “Cascade Lake”	172	3 days	none	none	No	68
bigmem	4	112	Intel “Sapphire Rapids”	2000	3 days	none	none	No	396
bigmem_intermediate	3	64	Intel “Ice Lake”	2000	14 days	none	none	No	396
gpu	36	64	Intel “Ice Lake”	990	3 days	none	none	Yes (4 A100/node)	396
gpu_h200	22	112	Intel “Sapphire Rapids”	990	3 days	none	none	Yes (4 H200/node)	843
intermediate	12	112	Intel “Sapphire Rapids”	990	14 days	none	none	No	396
unrestricted	8	48	Intel “Cascade Lake”	172	none	none	none	No	68
test	20	112	Intel “Sapphire Rapids”	990	12 Hours	5	112	No	396
gpu_test	12	64	Intel “Ice Lake”	487	12 Hours	2	64	Yes (8 A100 MIG 3g.20GB/node) – Limit 8 per job	172
serial_requeue	varies	varies	AMD/Intel	varies	3 days	none	none	No	varies
gpu_requeue	varies	varies	AMD/Intel	varies	3 days	none	none	Yes	varies
PI/Lab nodes	varies	varies	varies	varies	none	none	none	varies	varies

Partition Details

sapphire

The sapphire partition has a maximum run time of 3 days. Serial, parallel, and interactive jobs are permitted on this queue, and this is the most appropriate location for MPI jobs. This partition has 184 nodes connected by a NDR InfiniBand (IB) fabric, where each node configured with 2 Intel Xeon Sapphire Rapids CPUs, 990 GB of RAM, and 400 GB of local scratch space. Each Intel CPU has 56 Cores, and 100 MB of cache.

When submitting MPI jobs on the sapphire partition, it may be advisable to use the --contiguous option for best communication performance if your code is topology sensitive. Though all of the nodes are connected by Infiniband fabric, there are multiple switches routing the MPI traffic and Slurm will by default schedule you where ever it can find space. Thus your job may end up scattered across the cluster. The --contiguous option will ensure that the jobs are run on nodes that are adjacent to each other on the IB fabric. Be advised that using --contiguous will make your job pend longer, so only use it if you absolutely need it.

shared

The shared partition has a maximum run time of 3 days. Serial, parallel, and interactive jobs are permitted on this queue, and this is the most appropriate location for MPI jobs. This partition has 310 nodes connected by a HDR InfiniBand (IB) fabric, where each node configured with 2 Intel Xeon Cascade Lake CPUs, 172 GB of RAM, and 70 GB of local scratch space. Each Intel CPU has 24 Cores, and 48 MB of cache.

When submitting MPI jobs on the shared partition, it maybe advisable to use the --contiguous option for best communication performance if your code is topology sensitive. Though all of the nodes are connected by Infiniband fabric, there are multiple switches routing the MPI traffic and Slurm will by default schedule you where ever it can find space. Thus your job may end up scattered across the cluster. The --contiguous option will ensure that the jobs are run on nodes that are adjacent to each other on the IB fabric. Be advised that using --contiguous will make your job pend longer, so only use it if you absolutely need it.

bigmem

This partition should be used for large memory work requiring greater than 1000 GB RAM per job. Jobs requesting less than 1000 GB RAM are automatically rejected by the scheduler.

There is 3 day limit for work here. MPI or low memory work is not appropriate for the this partition, and inappropriate jobs may be terminated without warning. This partition has an allocation of 4 nodes with 1988 GB of RAM

bigmem_intermediate

This partition should be used for large memory work requiring greater than 1000 GB RAM per job. Jobs requesting less than 1000 GB RAM are automatically rejected by the scheduler. There is a minimum run time of 3 days and maximum run time of 14 days.

MPI or low memory work is not appropriate for the this partition, and inappropriate jobs may be terminated without warning. This partition has an allocation of 3 nodes with 2000 GB of RAM

gpu

This 36 node partition is for individuals wishing to use GPGPU resources. One will need to include #SBATCH --gres=gpu:n where n=1-4 in your SLURM submission scripts. Each node has 64 cores and is equipped with 4 x Nvidia A100s per node. See our GPU Computing section for more info on using and specifying GPU resources.

gpu_h200

This 22 node partition is for individuals wishing to use GPGPU resources. One will need to include #SBATCH --gres=gpu:n where n=1-4 in your SLURM submission scripts. Each node has 112 cores and is equipped with 4 x NVidia H200s per node. See our GPU Computing section for more info on using and specifying GPU resources.

intermediate

Serial and parallel (including MPI) jobs are permitted on this partition and this partition is intended for runs needing 3 to 14 days of runtime. This partition has an allocation of 12 nodes of the same configuration as above for the sapphire partition.

unrestricted

Serial and parallel (including MPI) jobs are permitted on this partition and 365 day limit on run time. Given this, there is no guarantee of 100% uptime. Running on this partition is done at the users own risk. Users should understand that if the queue is full it could take weeks or up to months for your job to be scheduled to run. unrestricted is made up of 8 nodes of the same configuration as above for the shared partition.

test

This partition is dedicated for interactive (foreground / live) work and for testing (interactively) code before submitting in batch and scaling. Small numbers (1 to 5) of serial and parallel jobs with small resource requirements (RAM/cores) are permitted on this partition; large numbers of interactive jobs or those requiring large resource requirements should really be done on another partition. Multiple partition submissions to this partition are forbidden (i.e. one is not permitted to do #SBATCH -p test,sapphire).

This partition is made up of 20 nodes of the same configuration as above for the sapphire partition. This smaller queue has a 12 hour maximum run time. This queue has a maximum of 112 cores and 1000 GB RAM. Jobs in this queue are not charged fairshare.

gpu_test

This 14 node partition is for individuals wishing to test GPGPU resources. One will need to include #SBATCH --gres=gpu:n where n=1-8 in your SLURM submission scripts. These nodes have 64 cores and are equipped with 4 x Nvidia A100s in Multi-Instance GPU (MIG) mode. Each GPU has two 3g.20GB MIG instances. This queue has a maximum of 2 jobs, 64 cores, 512 GB RAM, 8 MIG GPU’s, 12 hour run time. Users must request less than 8 CPUs/MIG GPU and 64GB/MIG GPU. This partition is intended for interactive, testing, and experimentation only. Multiple partition submissions to this partition are forbidden. See our GPU Computing section for more info on using and specifying GPU resources. Jobs in this queue are not charged fairshare.

serial_requeue

This partition is appropriate for single core (serial) jobs, jobs that require up to 8 cores for small periods of time (less than 1 day), or job arrays where each job instance uses less than 8 cores. Multinode jobs may be run in the partition but be advised that this is a heterogeneous partition and users are highly recommended to leverage the --constraint option to get a homogeneous block of compute and networking. The maximum runtime for this queue is 3 days. GPU jobs are rejected from this partition and should be run in gpu_requeue. As this partition is made up of an assortment of nodes owned by other groups in addition to the general nodes, jobs in this partition may be killed and requeued if a higher priority job (e.g. the job of a node owner) comes in.

Because serial_requeue takes advantage of slack time in owned partitions, times in the PENDING state can potentially be much shorter than the shared and sapphire partitions. Since jobs may be killed, requeued, and run a 2nd time, ensure that the jobs are a good match for this partition. For example, jobs that append output would not be good for serial_requeue unless the data files were zeroed out at the start to ensure output from a previous (killed) run was removed. Also, to ensure your job need not redo all its compute again, it is advisable to have checkpointing enabled for your code. We do advise that you use the --open-mode=append to see the requeue status/error messages in your log files. Without this option, your log files will be reset at the start of each (requeued) run, with no obvious indication of requeue events.

gpu_requeue

This partition is appropriate for gpu jobs that require small periods of time (less than 1 day). Multinode jobs may be run in the partition but be advised that this is a heterogeneous partition and users are highly recommended to leverage the --constraint option to get a homogeneous block of compute and networking. The maximum runtime for this queue is 3 days. One will need to include #SBATCH --gres=gpu:1 in your SLURM submission scripts to get access to this partition. As this partition is made up of an assortment of gpu nodes owned by other groups in addition to the public nodes, jobs in this partition may be killed but automatically requeued if a higher priority job (e.g. the job of a node owner) comes in.

Because gpu_requeue takes advantage of slack time in owned partitions, times in the PENDING state can potentially be much shorter than the gpu and gpu_h200 partitions. Since jobs may be killed, requeued, and run a 2nd time, ensure that the jobs are a good match for this partition. For example, jobs that append output would not be good for gpu_requeue unless the data files were zeroed out at the start to ensure output from a previous (killed) run was removed. Also, to ensure your job need not redo all its compute again, it is advisable to have checkpointing enabled for your code. We do advise that you use the --open-mode=append to see the requeue status/error messages in your log files. Without this option, your log files will be reset at the start of each (requeued) run, with no obvious indication of requeue events. See our GPU Computing section for more info on using and specifying GPU resources.

ITC, Kempner, HSPH, HUCE, and SEAS

For information on the partitions for these groups see:

Submitting Batch Jobs Using the `sbatch` Command

The main way to run jobs on the cluster is by submitting a script with the sbatch command. The command to submit a job is as simple as:

sbatch runscript.sh

The commands specified in the runscript.sh file will then be run on the first available compute node that fits the resources requested in the script. sbatch returns immediately after submission; commands are not run as foreground processes and won’t stop if you disconnect from the cluster.

When sbatch is run Slurm copies the current user environment and submission script into the scheduler. Thus the user is free to update their environment and the submission script they used. Note that this behavior does not apply to any thing else, so files, folders, executables, etc. will be executed and used as they are on disk the moment the script starts to use and access them, so do not update those files if you do not want those changed propagated. When the scheduler launches the script, the script will start in the directory the user submitted the job from.

A typical submission script, in this case loading a Python module and having Python print a message, will look like this:

NOTE: It is important to keep all #SBATCH lines together and at the top of the script; no comments, bash code, or variables settings should be done until after the #SBATCH lines. Otherwise, Slurm may assume it’s done interpreting and skip any that follow.

#!/bin/bash
#SBATCH -c 1                # Number of cores (-c)
#SBATCH -t 0-00:10          # Runtime in D-HH:MM, minimum of 10 minutes
#SBATCH -p serial_requeue   # Partition to submit to
#SBATCH --mem=100           # Memory pool for all cores (see also --mem-per-cpu)
#SBATCH -o myoutput_%j.out  # File to which STDOUT will be written, %j inserts jobid
#SBATCH -e myerrors_%j.err  # File to which STDERR will be written, %j inserts jobid

# load modules
module load python/3.10.9-fasrc01

# run code
python -c 'print("Hi there.")'

In general, a submission script is composed of 4 parts:

The #!/bin/bash line allows the script to be run as a bash script.
The #SBATCH lines which are instructions for Slurm.
Commands loading any necessary modules and setting any variables, paths, etc.
The execution line itself, in this case calling python and having it print a message.

The #SBATCH lines shown above set the following key parameters:

#SBATCH -c 1: Sets the number of cores (threads) that you’re requesting. Make sure that your tool can use multiple cores before requesting more than one. If this parameter is omitted, Slurm assumes -c 1. For more on parallel work see: threads, MPI
#SBATCH -t 0-01:00: Specifies the running time for the job in day-hour:minute (DD-HH:MM) format. Other acceptable time formats include “minutes”, “minutes:seconds”, “hours:minutes:seconds”, “days-hours”, and “days-hours:minutes:seconds”. If your job runs longer than the value you specify here, it will be canceled. Jobs have a maximum run time which varies by partition (see table above), though extensions can be done. There is no fairshare penalty for over-requesting time, though it will be harder for the scheduler to backfill your job if you overestimate.
#SBATCH -p serial_requeue: Specifies the Slurm partition under which the script will be run. See the partitions description above for more information. If you do not specify this parameter you will be given serial_requeue by default.
#SBATCH --mem=100: Specifies how much memory you require per node. Default units are MB, and users can use suffixes for other units [K|M|G|T]. Accurate specifications allow jobs to be run with maximum efficiency on the system. There are two main options, --mem-per-cpu and --mem. The --mem option specifies the total memory pool per node. If you must do work across multiple compute nodes (e.g. MPI code) and want to scale your memory allocation on a per core basis, then you should use the --mem-per-cpu option, as this will allocate the amount specified for each of the cores you’re requesting, whether it is on one node or multiple nodes. If this parameter is omitted, then you are granted 100 MB by default. Chances are good that your job will be killed as it will likely go over this amount, so one should always specify how much memory you require.
#SBATCH -o myoutput_%j.out: Specifies the file to which standard out will be appended. If a relative file name is used, it will be relative to your current working directory. The %j in the filename will be substituted by the JobID at runtime. If this parameter is omitted, any output will be directed to a file named slurm-JOBID.out in the current directory.
#SBATCH -e myerrors_%j.err: Specifies the file to which standard error will be appended. Slurm submission and processing errors will also appear in the file. The %j in the filename will be substituted by the JobID at runtime. If this parameter is omitted, any output will be directed to a file named slurm-JOBID.err in the current directory.
#SBATCH --test-only
While not shown above, adding this option to your script will tell the scheduler to return information on what would happen if you submit this job. This is a good and easy way to determine if you script is viable as well as give a rough estimate of how long it would take to schedule in the current queue load.
#SBATCH --account=some_lab
If you are in more than one lab, please ensure that you are charging your Fairshare to the appropriate group by using this option in all of your job scripts and specifying the lab group.

Other useful options not shown above are:

#SBATCH --gpus=1: Specifies how many gpus are needed for the computation. For more see the GPU specific section.
#SBATCH --test-only: Adding this option to your script will tell the scheduler to return information on what would happen if you submit this job. This is a good and easy way to determine if you script is viable as well as give a rough estimate of how long it would take to schedule in the current queue load.
#SBATCH --account=jharvard_lab: If you are in more than one lab, this option will charging your usagee to the appropriate group.

It should be noted that all options that are prefixed by #SBATCH can also be set on the command line and visa versa. For example if you wanted to set the partition via commandline instead you would do: sbatch -p PARTITIONNAME runscript.sh

Notifications by Email

The scheduler can send email to you for various job states (FAIL and END being the most useful). But please bear in mind that this must be used responsibly as one user can quickly overwhelm the mail system and affect the notifications of all users by clogging up the mail queue. Keep in mind that tens or even hundreds of thousands of jobs may be in flight at a given time. This is why below we will strongly caution against using the ALL mail type. If you are using a metascheduler, job arrays, or just many jobs, please try to avoid adding too much burden to the email queue; Sending hundreds or thousands of emails can cause email backups, not to mention fill up your inbox.

To add mail notification to your job script you can use the --mail-type option. You can find all the options available in the sbatch documentation. In addition if you specify END you will receive a summary of your job performance from jobstats.

The user to be notified is indicated with --mail-user. If no mail user is specified, Slurm uses the email address that is listed with your account.

Monitoring Job Progress

To monitor jobs use sacct. sacct with out any options will print out all the jobs you have run in the past day. sacct -j JOBID will show you a specific job. Note that sacct is almost live data, in addition the various accounting fields (such as memory usage) are incomplete until the job finishes. For monitoring live performance stats use the jobstats command. Slurm keeps past job records, so users can look back at their historic usage for up to 6 months. If you need data from further back contact FASRC to get access to our job archive.

sacct can provide much more detail as it has access to many of the resource accounting fields that SLURM uses. For example, to get a detailed report on the memory and CPU usage for an array job (see below for details about job arrays):

[jharvard@boslogin01 ~]? sacct -j 44375501 --format JobID,Elapsed,ReqMem,MaxRSS,AllocCPUs,TotalCPU,State   
JobID      Elapsed    ReqMem   MaxRSS AllocCPUS TotalCPU State
------------ ---------- --------- ------- ---------- ---------- ----------
44375501_[1+ 00:00:00   40000Mc           8    00:00:00   PENDING
44375501_1   2-03:50:53 40000Mc           8    2-03:50:23 COMPLETED
44375501_1.+ 2-03:50:53 40000Mc 34372176K 6    2-03:50:23 COMPLETED
44375501_1.+ 2-03:50:53 40000Mc 1236K     8    00:00.004  COMPLETED
44375501_2   1-23:47:35 40000Mc           8    1-23:47:18 COMPLETED
44375501_2.+ 1-23:47:35 40000Mc 34467196K 6    1-23:47:17 COMPLETED
44375501_2.+ 1-23:47:36 40000Mc 1116K     8    00:00.003  COMPLETED
44375501_3   1-23:32:36 40000Mc           8    1-23:32:15 COMPLETED
44375501_3.+ 1-23:32:36 40000Mc 34389040K 6    1-23:32:15 COMPLETED
44375501_3.+ 1-23:32:37 40000Mc 1224K     8    00:00.004  COMPLETED
44375501_4   1-21:59:30 40000Mc           8    1-21:59:07 COMPLETED
44375501_4.+ 1-21:59:30 40000Mc 34389044K 6    1-21:59:07 COMPLETED

The jobstats and seff-account commands are summary commands based off the data in sacct.

Slurm provides information about the job State. This value will typically be one of PENDING, RUNNING, COMPLETED, CANCELLED, or FAILED.

PENDING	Job is awaiting a slot suitable for the requested resources. Jobs with high resource demands may spend significant time PENDING.
RUNNING	Job is running.
COMPLETED	Job has finished and the command(s) have returned successfully (i.e. exit code 0).
CANCELLED	Job has been terminated by the user or administrator using scancel.
FAILED	Job finished with an exit code other than 0.

To learn more detailed information about individual jobs that are in the PENDING or RUNNING you can run the scontrol command. For example:

[jharvard@boslogin06 general]# scontrol show job 7000364
JobId=7000364 JobName=run_pros
UserId=jharvard(21442) GroupId=jharvard_lab(10483) MCS_label=N/A
Priority=313513 Nice=0 Account=jharvard_lab QOS=normal
JobState=PENDING Reason=Resources Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
RunTime=00:00:00 TimeLimit=04:00:00 TimeMin=N/A
SubmitTime=2026-04-21T05:51:54 EligibleTime=2026-04-21T05:51:54
AccrueTime=2026-04-21T05:51:54
StartTime=2026-04-22T00:20:00 EndTime=2026-04-22T04:20:00 Deadline=N/A
SuspendTime=None SecsPreSuspend=0 LastSchedEval=2026-04-21T09:28:15 Scheduler=Main
Partition=sapphire,shared AllocNode:Sid=holylogin06:928788
ReqNodeList=(null) ExcNodeList=(null)
NodeList= SchedNodeList=holy8a24607
StepMgrEnabled=Yes
NumNodes=1-1 NumCPUs=32 NumTasks=32 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
ReqTRES=cpu=32,mem=250G,node=1,billing=36
AllocTRES=(null)
Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
MinCPUsNode=1 MinMemoryCPU=8000M MinTmpDiskNode=0
Features=(null) DelayBoot=00:00:00
OverSubscribe=OK Contiguous=0 Licenses=(null) LicensesAlloc=(null) Network=(null)
Command=/n/netscratch/jharvard_lab/Lab/jharvard/run_BlueJay.sh
SubmitLine=sbatch run_BlueJay.sh
WorkDir=/n/netscratch/jharvard_lab/Lab/jharvard/finaladd5perc_v28
StdErr=
StdIn=/dev/null
StdOut=/n/netscratch/jharvard_lab/Lab/jharvard/finaladd5perc_v28/slurm-7000364.out

Of particular interest will be the Reason and StartTime fields. The Reason field will state why the job is pending, while the StartTime will give the current best estimate based on current cluster state as to when the job will start. Note that for job arrays this command will print out all elements, so it is best to specify which element you are interested in.

See the Broader Queue

The showq command can be used to show what the rest of the partition looks like. Often your job is pending due to other people in the partition. The showq command then shows you an overview of all the jobs for a specific partition. showq is invoked by doing:

showq -o -p PARTITIONNAME

Where -o orders the pending queue by priority, with the next job to be scheduled at the top. -p specifies the partition that you want to look at.

The sinfo command is used to get the general state of nodes in a partition. Nodes can be in the following states:

IDLE	Node is available for work.
MIXED	Node is partially used.
ALLOCATED	Node is fully used.
COMPLETING	Node has jobs which are finishing up.
PLANNED	Node will be used by a future job.
RESERVED	Node is part of a Reservation.
DRAINING	Node is closed to new jobs and existing jobs will run to completion.
DOWN	Node is offline.

You can then use scontrol show node NODENAME to get information on a given node including why it may be DOWN or DRAINING.

Canceling Jobs

If for any reason, you need to cancel a job that you’ve submitted, just use the scancel command with the job ID.

scancel JOBID

If you don’t keep track of the job ID returned from sbatch, you should be able to find it with the sacct command described above. scancel can also do bulk cancellations based on various parameters such as Job Name and Partition.

Interactive Jobs and `salloc`

Though batch submission is the best way to take full advantage of the compute power the cluster, foreground/interactive jobs can also be run. These can be useful for things like:

Iterative data exploration at the command line
RAM intensive graphical applications like MATLAB or SAS
Interactive “console tools” like R and Jupyter
Significant software development and compiling efforts

There are two main types of interactive sessions: Graphical User Interface (GUI) and Command Line Interface (CLI). For graphical sessions FASRC provides Open OnDemand (OOD). With Open OnDemand a user can launch a job which will start a Remote Desktop on the cluster or some other application in OOD.

Command line interactive jobs are instead launched directly from the login nodes using salloc. Please note that salloc is disabled on FASSE due to security considerations, you will want to use FASSE OOD instead. salloc has all the same options as sbatch. To start an interactive session run:

salloc -p test -c 1 --mem=4G -t 0-6:00:00

This will ask for 1 core and 4GB of memory on the test partition for 6 hours. With salloc if you append a command it will run it and then exit (this includes /bin/bash which will just exit), but if you append no command it will simply start a remote shell on the node the scheduler selects. Jobs submitted via salloc behave like normal jobs for the sake of scheduling, as such salloc may hang for a while if the partition you select is busy. As such it is wise to select a partition like test or gpu_test where you are guaranteed immediate access. If you intend to use a busy partition, we recommend switching to using Open OnDemand Remote Desktop.

Command line interactive sessions require you to be active in the session. If you go more than an hour without any kind of input, it will assume that you have left the session and will terminate it. If you have interactive tasks that must stretch over days, we recommend switching to Open OnDemand Remote Desktop.

Software

Collapse

Users are permitted to install whatever software relevant to their research on the cluster, provided it complies with our Acceptable Use Policy. FASRC clusters run a unified Operating System (Rocky Linux 8) and system architecture (x86-64), so software built on one system should generally work on the entire cluster (unless built against a specific hardware type). Users are responsible for managing and maintaining their own software stack. Under no circumstances will a user be given sudo access to install software. See the software guide for more on how to use FASRC provided software modules, how to use Podman or Singularity containers, and how to install software of various types.

Using GPUs

To request a single GPU on slurm just add #SBATCH --gpus=1 to your submission script and it will give you access to a GPU. For more on GPU computing see our more in depth GPGPU Document.

Specifying GPU Type

For users who wish to specify which type of GPU they wish to use, especially for those using heterogeneous partitions like gpu_requeue, there are two methods that can be used. The first is using --constraint="<tag>", this will constrain the job to only run on gpus of a certain class. A full listing of constraints can be found below. The second method is defining the specific model you want using --gpus=<model>:1. For example if you want a A100 with 80GB of onboard memory then you would specify --gpus=nvidia_a100-sxm4-80gb:1.

a100

nvidia_a100-sxm4-40gb: Nvidia A100 SXM4 40GB
nvidia_a100-sxm4-80gb: Nvidia A100 SXM4 80GB

h100 & h200

nvidia_h100_80gb_hbm3: Nvidia H100 80GB HBM3
nvidia_h200: Nvidia H200 140GB

mig

nvidia_a100_1g.5gb: Nvidia A100 1g MIG 5GB
nvidia_a100_1g.10gb: Nvidia A100 1g MIG 10GB
nvidia_a100_3g.20gb: Nvidia A100 3g MIG 20GB

a40

nvidia_a40: Nvidia A40 40GB

rtx

nvidia_rtx_a6000: Nvidia RTX A6000 PCIe 48GB
nvidia_rtx_pro_6000_blackwell_server_edition: Nvidia RTX PRO 6000 Blackwell 96GB

Some of the GPUs listed here were purchased by specific groups and only available via gpu_requeue. To find out what specific types of gpu’s are available on a partition run scontrol show partition <PartitionName> and look under the TRES category.

Parallelization

Using Threads such as OpenMP

One of the basic methods for parallelization is to use a threading library, such as pthreads, OpenMP, or applications that use OpenMP under the hood (e.g. numpy, OpenBLAS). Slurm by default does not know what cores to assign to what process it runs, in addition for threaded applications you need to make sure that all the cores you request are on the same node. Below is an example script that both ensures all the cores are on the same node, and lets Slurm know which process gets the cores that you requested for threading.

#!/bin/bash
#SBATCH -c 8 # Number of threads
#SBATCH -t 0-00:30:00 # Amount of time needed DD-HH:MM:SS
#SBATCH -p sapphire # Partition to submit to
#SBATCH --mem-per-cpu=100 #Memory per cpu
module load intel/25.3.1-fasrc01
srun -c $SLURM_CPUS_PER_TASK MYPROGRAM > output.txt 2> errors.txt

The most important aspect of the threaded script above is the -c option which tells Slurm how many threads you intend to run with. If you are using OpenMP you will want notify it of how many threads it can use by setting OMP_NUM_THREADS before the executable:

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

Using MPI

MPI (Message Passing Interface) is a standard that supports communication between separate processes, allowing parallel programs to simulate a large common memory space. OpenMPI, MPICH, and Intel MPI are available as modules on the cluster. As described in the module documentation, MPI libraries are a special class of module, called “Comp”, that is compiler dependent. To load an MPI library, load the compiler first.

module load intel/25.3.1-fasrc01 openmpi/5.0.10-fasrc01

Once an MPI module is loaded, applications built against that library are made available. This dynamic loading mechanism prevents conflicts that can arise between compiler versions and MPI library flavors.

An example MPI script with comments is shown below:

#!/bin/bash
#SBATCH -n 128 # Number of cores
#SBATCH -t 10 # Runtime in minutes
#SBATCH -p sapphire # Partition to submit to
#SBATCH --mem-per-cpu=100 # Memory per cpu in MB (see also --mem)
module load intel/25.3.1-fasrc01 openmpi/5.0.10-fasrc01
module load MYPROGRAM
srun -n $SLURM_NTASKS --mpi=pmix MYPROGRAM > output.txt 2> errors.txt

There are a number of important aspects to an MPI SLURM job.

Most partitions have a unified Infiniband fabric except for the requeue partitions. If you use the requeue partitions you will want to specify a IB fabric via the constraint option.
Memory should be allocated with the --mem-per-cpu option instead of --mem so that memory matches core utilization.
The -np option for mpirun or mpiexec (when these runners are used) should use the bash variable $SLURM_NTASKS so that the correct number of cores is passed to the MPI engine at runtime.
If network topology and communications overhead is a concern for your code, try using the --contiguous option which will ensure that all the cores you get will be adjacent to each other. Use this with caution though as it will make your job pend longer, as finding contiguous blocks of compute is difficult. Verify that the boost in performance is worth the extra wait time in the queue. If you do not include this option you will be given cores and what ever nodes that Slurm can find, which may be scattered across the cluster. Depending on your code this may or may not be a concern. Test your code in both modes to see if it is an option that is worth including if you don’t know off hand. It may not be worth including --continguous as the aggregate time of waiting plus runtime may be longer with --contiguous. The sbatch and srun documentation have more information on various fine tuning options.
The application must be MPI-enabled. Applications cannot take advantage of MPI parallelization unless the source code is specifically built for it.

Job Arrays

SLURM allows you to submit a number of “near identical” jobs simultaneously in the form of a job array. To take advantage of this, you will need a set of jobs that differ only by an “index” of some kind.

For example, say that you would like to run tophat, a splice-aware transcript-to-genome mapping tool, on 30 separate transcript files named trans1.fq, trans2.fq, trans3.fq, etc. First, construct a SLURM batch script, called tophat.sh, using special SLURM job array variables:

#!/bin/bash
#SBATCH -J tophat # A single job name for the array
#SBATCH -c 1 # Number of cores
#SBATCH --array=1-30 # Array range
#SBATCH -p serial_requeue # Partition
#SBATCH --mem 4000 # Memory request (4Gb)
#SBATCH -t 0-2:00 # Maximum execution time (D-HH:MM)
#SBATCH -o tophat_%A_%a.out # Standard output
#SBATCH -e tophat_%A_%a.err # Standard error

source activate tophat
tophat /n/netscratch/informatics_public/ref/ucsc/Mus_musculus/mm10/chromFatrans"${SLURM_ARRAY_TASK_ID}".fq

The --array flag sets the number of elements to be run. Each array element is treated by the scheduler as an independent job for the sake fo scheduling.

In the script, two types of substitution variables are available when running job arrays. The first, %A and %a, represent the job ID and the job array index, respectively. These can be used in the sbatch parameters to generate unique names. The second, SLURM_ARRAY_TASK_ID, is a bash environment variable that contains the current array index and can be used in the script itself. In this example, 30 jobs will be submitted each with a different input file and different standard error and standard out files. More detail can be found on the SLURM job array documentation page and our Submitting Large Numbers of Jobs page.

Checkpointing

Slurm does not automatically checkpoint, i.e. create files that your job can restart from. To protect against job failure (due to code error or node failure) and to allow your job to be broken up into smaller chunks it is always advisable to checkpoint your code so it can restart from where it left off. This is especially valuable for jobs on partitions subject to requeue, but is also just generally useful for any type of job. Checkpointing varies from code type to code type and needs to be implemented by the user as part of their code base. Some resources for checkpointing codes that do not have them built-in include Distributed MultiThreaded CheckPointing (DMTCP) and Checkpoint/Restore in Userspace (CRIU).

Job dependencies

Many scientific computing tasks consist of serial processing steps. A genome assembly pipeline, for example, may require sequence quality trimming, assembly, and annotation steps that must occur in series. Launching each of these jobs without manual intervention can be done by repeatedly polling the controller with sacct until the State is COMPLETED. However, it’s much more efficient to let the SLURM controller handle this using the --dependency option.

[jharvard@boslogin01 examples]? sbatch assemble_genome.sh
Submitted batch job 53013437
[jharvard@boslogin01 examples]? sbatch --dependency=afterok:53013437 annotate_genome.sh
[jharvard@boslogin01 examples]?

When submitting a job, specify a combination of “dependency type” and job ID in the --dependency option. afterok is an example of a dependency type that will run the dependent job if the parent job completes successfully (state goes to COMPLETED). The full list of dependency types can be found on the SLURM doc site in the man page for sbatch. It is best not to create a chain of dependencies that is greater than 2-3 levels. Any more than that and the scheduler will become significantly slower. Dependencies should only be used if the resource requirements between each step are significantly different, or if you need to wait for an array to complete before you run a single job that processes all the array results. Be sure to think about whether you truly need dependencies or not.

Job Constraints

Sometimes, especially on the requeue partitions, jobs need to be constrained to run on specific hardware. Many times this is due to either the code being compiled for a specific architecture or because the code runs more efficiently on a specific type of host. Slurm provides for this functionality via the --constraint option (see the sbatch documentation for usage details). The features for constraint are defined by FASRC and fall into three broad categories: Processor, GPU, and Network. You can match against multiple of these but keep in mind the more constraints you use the longer your job will pend for as the scheduler will find it more difficult to find nodes that fit your needs. A list of the features available on the cluster follows, you can also see the features for a specific node by doing scontrol show node NODENAME.

Processor

amd: All AMD processors
intel: All Intel processors
avx: All processors that are AVX capable
avx2: All processors that are AVX2 capable
avx512: All processors that are AVX512 capable
milan: AMD Milan chips
genoa: AMD Genoa chips
turin: AMD Turin chips
skylake: Intel Skylake chips
sapphirerapids: Intel Sapphire Rapids
cascadelake: Intel Cascade Lake chips
icelake: Intel Ice Lake chips

GPU

To specify a GPU model, for example, A100 with 80GB refer to Specifying GPU Type

rtxa6000: Nvidia RTX A6000 GPU
rtx6000pro: Nvidia RTX PRO 6000 Blackwell GPU
a40: Nvidia A40 GPU
a100: Nvidia A100 GPU
a100-mig: Nvidia A100 GPU MIG
h100: Nvidia H100 GPU
h200: Nvidia H200 GPU

Network

holyhdr: Holyoke HDR Infiniband Fabric
holyndr: Holyoke NDR Infiniband Fabric

Fairshare and Job Prioritization

We use a multifactor method of job scheduling on the cluster. Job priority is assigned by a combination of fair-share and length of time a job has been sitting in the queue. You can find out the priority calculation for your jobs by using the sprio command, such as sprio -j JOBID.

Fairshare is shared on a lab basis, so usage by any member of the lab will impact the score of the whole lab as the lab is pulling from a common pool. Fairshare has a 3 day halflife and naturally recovers if your lab does not run any jobs. Thus it is wise to store up fairshare if you need to do significant runs, and plan your runs accordingly in order to maintain a good fairshare score. You can learn more about your fairshare score and slurm usage by using the sshare command, such as sshare -U which shows your current score.

The other factor in priority is how long you have been sitting in the queue. The longer your job sits in the queue the higher its priority grows, out to a maximum of 3 days. If everyone’s priority is equal then FIFO (first in first out) is the scheduling method. We weight the age of a job that has pended for 3 days to be equal to a fairshare score of 0.1.

We also have backfill turned on. This allows for jobs which are smaller to sneak in while a larger higher priority job is waiting for nodes to free up. If your job can run in the amount of time it takes for the other job to get all the nodes it needs, SLURM will schedule you to run during that period. This means knowing how long your code will run for is very important and must be declared if you wish to leverage this feature. Otherwise the scheduler will just assume you will use the maximum allowed time for the partition when you run. The better your constrain your job in terms of CPU, Memory, and Time the easier it will be for the backfill scheduler to find you space and let your job jump ahead in the queue.

For more see:

Troubleshooting

A variety of problems can arise when running jobs on the cluster. Many are related to resource misallocation, but there are other common problems as well.

Error	Likely cause
`JOB <jobid> CANCELLED AT <time> DUE TO TIME LIMIT`	You did not specify enough time in your batch submission script. The `-t` option sets time in minutes or can also take `D-HH:MM` form (`0-12:30` for 12.5 hours)
`Job <jobid> exceeded <mem> memory limit, being killed`	Your job is attempting to use more memory than you’ve requested for it. Either increase the amount of memory requested by `--mem` or `--mem-per-cpu` or, if possible, reduce the amount your application is trying to use. For example, many Java programs set heap space using the `-Xmx` JVM option. This could potentially be reduced. For jobs that require truly large amounts of memory (>1 Tb), you may need to use the `bigmem` SLURM partition. Genome and transcript assembly tools are commonly in this camp.
`SLURM_receive_msg: Socket timed out on send/recv operation`	This message indicates a failure of the SLURM controller. Though there are many possible explanations, it is generally due to an overwhelming number of jobs being submitted, or, occasionally, finishing simultaneously. If you want to figure out if SLURM is working use the `sdiag` command. `sdiag` should respond quickly in these situations and give you an idea as to what the scheduler is up to.
`JOB <jobid> CANCELLED AT <time> DUE TO NODE FAILURE`	This message may arise for a variety of reasons, but it typically indicates that the host on which your job was running can no longer be contacted by SLURM. Jobs that die from NODE_FAILURE are automatically requeued by the scheduler.

Harvard T.H. Chan School of Public Health – FAS Research Computing Overview

Introduction

The Harvard T.H. Chan School of Public Health (HSPH) is using the FAS Research Computing environment to host data and run the analysis. If you are curious, take a look at the HCSPH & FAS Research Computing Q&A page for additional information. A list of the currently installed software on the cluster is available online. Please see our document on installing software yourself.

Requesting an account

Requesting a Research Computing account is a semi-automated process, requiring a selection of a PI (your PI must have an existing FASRC account) as your sponsor and whose lab you will belong to. The sponsor PI will then approve or deny the request. Once the PI has approved the request, turnaround is generally within the hour (Outline of Process). If the request is for an EXTERNAL user, this may take slightly longer as external users must be vetted manually.

After receiving your credentials, you must attend one of our monthly New User Trainings or watch our Introduction to the Cluster videos if you selected cluster access (e.g. – the ability to run jobs). Training is essential for the proper use of our complex systems. The email stating your account has been created will have links to the necessary getting started and running jobs documentation necessary to get started on the cluster.

NOTE: If you did not request cluster access, you will not have a home directory or be able to run jobs. As such, you are not required to take the cluster quiz but will still need to set your password and acquire your OpenAuth token as shown in the getting started docs.

Important: Regarding Account Sharing – The sharing of accounts is a violation of Harvard and RC information security policies. You cannot share your account with another individual, nor should anyone else know your login credentials. Additionally, you may have only one account. Please do not sign up for multiple accounts.

Cluster Access

An account with cluster access provides users with access to resources hosted in the FAS RC environment, including expert consulting help coupled with extensive resources, such as over 100 PB of storage, over 110,000 processing cores, as well as numerous software modules and applications. For further details on accessing the cluster, please see the FAQ or our helpful Access and Login guide.

Billing and Usage

The previous annual per-user charge for FAS Research Computing accounts is no longer used as of the fiscal year 2019. Standard services, including accounts and access to the FAS RC high-performance computing cluster, will be effectively free to all Harvard Chan School community members via the account of an eligible PI (See: HCSPH PI Eligibility Policy). These FASRC computing costs are now covered by the school’s overhead rate. Supplemental resources such as increased storage and VMs still involve additional charges, please see our Billing FAQ.

Authentication and Security

FAS Research Computing uses a two-factor authentication system. The OpenAuth client application exists for Windows, OSX, and Linux, or you can import your RC token into Google Authenticator or DUO to display the necessary auth code there. Each account and token is unique to the individual account holder. Please note that the sharing of accounts/credentials is a violation of Harvard and RC security policies. In order to access some of the web servers, storage, and other services at FAS Research Computing, whether on wired or wireless, you will likely need to use a VPN connection. Again, consult our Access and Login guide for more details.

Using The Cluster

Taking advantage of the cluster’s massive compute capabilities is easy if you’ve read through our Quick Start Guide and Running Jobs document. You’ll find many more helpful documents in our online documentation. Additional support for using the FASRC is available via training and weekly virtual office hours. Additional computational training related to research computing is available from the Harvard Chan School Bioinformatics Core.

HCSPH has their own partitions of purchased hardware on the cluster. These partitions are open to HSPH users and their allocation is governed by the relative fairshare of each group. These partitions are:

hsph: This block of 6272 Intel Sapphire Rapids cores. Each node is water-cooled and contains 112 cores, and 1TB of RAM. The nodes are interconnected with NDR Infiniband. This partition has a 3 day time limit.
hsph_gpu: This block of GPU’s contains 128 AMD Genoa cores and 8 Nvidia H200 GPUs. Each node is water-cooled and contains 64 cores, 1.5TB of RAM, and 4 GPU’s. The nodes are interconnected with NDR Infiniband. This partition has a 3 day time limit.

Resource planning

Note that it is relatively easy to overload the file storage system. Please use our high-performance scratch storage filesystems for high I/O jobs. Also, if you are submitting a large number of tasks, please see our Submitting Large Numbers of Jobs document, and please try to keep your jobs efficient and bundle them in 6 to 10 minute lots. If in doubt, contact us.

Interactive sessions

The cluster allows interactive use which is great for exploring new tools or running shell-based sessions (SAS, MATLAB, R) without having to submit jobs. Please see our Interactive Sessions information and Open OnDemand documentation.

Data storage and security

Data can be transferred to and from the cluster using multiple file transfer methods. Please see our Getting Started with FASRC Storage documentation. Certain HCSPH researchers have access to the FAS Research Computing computing file systems, in particular each existing PI with an account and lab group with cluster access RC is eligible for 4TB of base lab share to facilitate basic research workflow. Additional storage can be purchased: See our Data Storage services page.

Special Security Requirements

The general FAS RC cluster storage environment is not suitable for storing data with special security requirements. If access to your data needs to be limited in any way, contact us prior to transferring data. Important: Do not store data which is covered under a data use agreement or is otherwise considered high risk without consulting FAS RC first. Important: The sharing of accounts is a violation of Harvard and RC information security policies. Please contact us if you have a special need.

Contact and support

FAS RC has a number of methods for supporting researchers in need of help. Please see our FASRC Support page for details. You may also contact the HCSPH Bioinformatics Core for local help. Additionally, FAS RC hosts Virtual Office Hours weekly.

LOGIN AND AUTHENTICATION

My login is slow or my batch commands are slow

My alternate shell (csh, tcsh, etc.) doesn’t work right

SSH key error, DNS spoofing message

SFTP exits after a few seconds

What happens to my account when I leave/graduate?

How do I request membership in additional lab groups?

Can I use SSH keys to log in without a password?

How do I get a Research Computing account?

Can someone else approve my account request?

Can I share an account? – Account Security Policies

How do I login to the FASRC cluster?

How do I reset my Research Computing account password?

How do I unlock my locked Research Computing account?

How do I install and launch OpenAuth?

How do I logon to the Research Computing VPN?

I need an AWS account and/or Amazon AWS virtual machine

FILESYSTEMS AND AUTHORIZATION

Where is ftp?

How do I request membership in additional lab groups?

What’s the best way to transfer my data?

How do I access my cluster home directory from my laptop?

How do I check how much space I’ve used, what’s my quota?

I accidentally deleted my data, how do I get it back?

Why are all my files executable?

Why does my UMASK not work?

Is my home directory available as a network filesystem share?

I am seeing weird errors about file locking with HDF5. What do I do?

SOFTWARE

I need cluster access to Gaussian

I need to download Geneious Pro or MOE (only available for FAS users)

I can’t search for R

Where is FTP?

How do I load a module or software on FASRC cluster?

FileZilla: I have to enter my OpenAuth code every 30 seconds

Git/Github: 403 Forbidden while accessing https://github.com…

How do I run a Matlab script on the FASRC cluster?

Perl modules: Can’t locate XX.pm in @INC

Illegal Instruction

Installing LaTeX packages

JOBS AND SLURM

How do I know what partitions I have access to?

How do I know what memory limit to put on my job?

How do I figure out how efficient my job is?

Will single core/thread jobs run faster on the cluster?

My login is slow or my batch commands are slow

How do I request membership in additional lab groups?

Can I query SLURM programmatically?

Are their policies or guidelines for using the cluster responsibly?

How do I submit a batch job to the FASRC cluster queue with SLURM?

How do I submit an interactive job on the cluster?

How do I view or monitor a submitted job?

My job is PENDING. How can I fix this?

SLURM Errors: Job Submission Limit (per user)

SLURM Errors: Device or resource busy

SLURM errors: Job cancelled due to preemption

SLURM Errors: Memory limit

SLURM Errors: Node Failure

SLURM errors: Socket timed out. What?

SLURM Errors: Time limit

What is Fair-Share?

Can I send mail from the cluster?

I see dummy4XD jobs, but I didn’t submit them?

I see nodes marked as DRAINING, DOWN, or COMPLETING in the partition that I am using what can I do?

BILLING

VDI (Open OnDemand)

Why is my Jupyter notebook VDI session terminated right after it starts?

Home Directories

Lab Directories

Introduction

Getting Started

Storage and Scratch on the Cluster

Slurm Documentation

Summary of Slurm Commands

Slurm Global Limits and Defaults

Slurm Partitions

Partition Details

sapphire

shared

bigmem

Why does my `UMASK` not work?

Submitting Batch Jobs Using the `sbatch` Command

Interactive Jobs and `salloc`