\nExceptions<\/strong>: The - Any owned nodes that are not available will be cleaned up after we address the above partitions.\u00a0 The cluster and its login nodes are now on Rocky Linux 8. While this is a CentOS derivative, there are some differences from CentOS 7 that could impact your jobs. Please see our Rocky 8 Transition Guide for more\u00a0information:\u00a0https:\/\/docs.rc.fas.harvard.edu\/kb\/rocky-8-transition-guide\/<\/a><\/p>\n Much has changed with the software offerings on the FASRC cluster. These changes put us more inline with how peer institutions handle software as well as reduces\u00a0the sheer volume and complexity of modules and versions that FASRC was having to maintain. There are now fewer modules<\/a>, but more resources and tools for building and installing your\u00a0own software.\u00a0Please see: https:\/\/docs.rc.fas.harvard.edu\/kb\/rocky-8-transition-guide\/<\/a> and https:\/\/docs.rc.fas.harvard.edu\/kb\/software\/<\/a><\/p>\n After an analysis (https:\/\/www.rc.fas.harvard.edu\/blog\/cluster-fragmentation<\/a>\/) of run time which found that over 95% of jobs complete within 3 days , we will be changing all the public partitions on Cannon, excluding unrestricted, to a 3 day time limit.\u00a0To accommodate the 5% of jobs longer than 3 days we will be adding an \"intermediate\" partition for jobs that need to run between 3 to 14 days; with \"unrestricted\" handling the rest. Also for new partitions owned by specific groups we will be instituting a default 3 day time limit, existing partitions are not impacted and groups may change this default to suit their needs.<\/p>\n You will find new training opportunities as well as new user sessions at: https:\/\/www.rc.fas.harvard.edu\/upcoming-training\/<\/a> If you load modules in your .bashrc or other startup script, you will likely need to modify this as some modules will no longer be available or will have changed.\u00a0 You _should not_ be loading the old centos6 module as it is full deprecated.<\/p>\n Logging in via SSH after June 9th you will likely see this error: Due to the OS and other changes\u00a0under the hood, Conda\/Anaconda Python environments will need to be rebuilt. Running environments built under CentOS 7 may produce unexpected results or fail.<\/p>\n Please note that Conda has been replaced with Mamba, a faster but fully Conda command-compatible replacement.<\/p>\n See:\u00a0https:\/\/docs.rc.fas.harvard.edu\/kb\/python\/<\/a> and\u00a0 https:\/\/docs.rc.fas.harvard.edu\/kb\/rocky-8-transition-guide\/\u00a0<\/a><\/p>\n We have moved away from the old pkill process on login nodes which killed processes using too much CPU or memory (RAM)\u00a0usage to try and maintain a fair balance for all. The Rocky 8 login nodes use a system-level mechanism called cgroups\u00a0which limits each logged-in account to 1 core and 4GB of memory.\u00a0 Regarding OOD\/VDI: The Open OnDemand instances for Cannon (not FASSE) will involve a change to each user's settings and the location of the folder containing those settings and other OOD-specific files. This also means that values you had filled out in the OOD start form will not be retained. Form values will revert to the defaults.<\/p>\n The new location in your Cannon home directory will be ~\/.fasrcood If you have any further concerns or unanswered questions\u00a0please visit office hours (https:\/\/www.rc.fas.harvard.edu\/training\/office-hours\/<\/a>) or submit a help ticket (https:\/\/portal.rc.fas.harvard.edu\/rcrt\/submit_ticket<\/a>) and we will do our best to help.\u00a0Also please consider\u00a0attending one of the software trainings\u00a0if your concern is software-related.\u00a0https:\/\/www.rc.fas.harvard.edu\/upcoming-training\/<\/a><\/p>\n \nThanks, <\/p>\n The annual multi-day power downtime at MGHPCC is on-going.\u00a0 June 5th-8th with return to full service 9AM Friday June 9th.<\/p>\n During this downtime we will be performing major updates to the cluster's operating system, public partition time limits, and significant changes to our software offerings<\/a> and procedures. This will affect ALL cluster users and as described below may require you to modify your normal workflows and software chains. \n- All running and all pending jobs will be purged Monday morning June 5th. All pending jobs or unfinished jobs will need to be re-submitted once the cluster is back up.<\/p>\n \n- All login nodes will be down for the duration and will have their OS updated to Rocky 8. Once complete, the login process will not change, but the underlying operating system will. Logging in via SSH on or after June 9th, you will likely see \n- All storage at MGHPCC\/Holyoke will be powered down during the outage. Boston storage servers will not be powered down but will likely be unavailable as there will be no nodes available to access them. Some Boston Samba shares may be accessible, but please assume they will also be affected at various times.<\/p>\n <\/p>\n Currently the cluster nodes (Cannon and FASSE), and indeed most of our infrastructure, are built on CentOS 7, the non-commercial version of RedHat's Enterprise Linux. CentOS is being discontinued by RedHat and new development ceased at the end of 2021. As a result, we are moving to Rocky 8 Linux, created by the same people who began CentOS and which much of the HPC community is also getting behind. Given the wide adoption indicated by other HPC sites, we feel confident Rocky Linux is the right choice so that we will be part of a large community where we can both find common support and contribute back.<\/p>\n Rocky Linux will be a major update with significant changes in the same way that our move from CentOS 6 to CentOS 7 was.\u00a0 As such, there will be issues that will affect some pipelines, software, or codes. Additionally, we will necessarily be revamping our software offerings and, concurrently, also giving end-users more control over their own software with new build tools such as Spack.\u00a0<\/p>\n Rocky 8 Transition Guide: https:\/\/docs.rc.fas.harvard.edu\/kb\/rocky-8-transition-guide\/<\/p>\n Please DO expect to see SSH warnings when you log in again after June 8th as all machines including login nodes will have been completely rebuilt. See the FAQ at: https:\/\/docs.rc.fas.harvard.edu\/kb\/rocky-8-transition-guide\/#FAQ<\/a><\/p>\n Regarding OOD\/VDI: The Open OnDemand instances for Cannon (not FASSE) will involve a change to each user's settings and the location of the folder containing those settings and other OOD-specific files. This also means that values you had filled out in the OOD start form will not be retained. Form values will revert to the defaults.<\/p>\n The new location in your Cannon home directory will be ~\/.fasrcood<\/p>\n Any \"development mode\" AKA \"sandbox\" apps that you have installed to ~\/fasrc\/dev (approximately 78 of you have created these) will no longer be visible through your OOD Dashboard, and will need to be moved to ~\/.fasrcood\/dev . The old folder at ~\/fasrc will no longer be used and can be removed after June 8th.<\/p>\n The location for FASSE settings in your home directory will not change and will remain at ~\/.fasseood<\/p>\n During the summer of 2022 we did an analysis of job run times on the Cannon cluster with the goal of reassessing our existing partition time limits (which are 7 days). A reduced time limit has many benefits such as reduced cluster fragmentation<\/a>, lower wait times, and short times to drain nodes for service.\u00a0 As a result of this analysis we found that over 95% of jobs complete within 3 days on all the public partitions excluding unrestricted.<\/p>\n We will be changing all the public partitions on Cannon, excluding unrestricted, to a 3 day time limit. To accommodate the 5% of jobs longer than 3 days we will be adding an intermediate partition for jobs that need to run between 3 to 14 days; with unrestricted handling the rest. Also for new partitions owned by specific groups we will be instituting a default 3 day time limit, existing partitions are not impacted and groups may change this default to suit their needs.<\/p>\n We will reduce our dependence on the Lmod modules and provide end-users with more and expanded options for building and deploying the software<\/a> they need. We will incorporate the use of Spack<\/a> to give users more power to build the software and versions they need. Those who came to FASRC from other HPC sites may recognize this as the norm at many sites these days.\u00a0<\/p>\n There will be a Singularity image to allow those who need it to run a CentOS 7 image until they can transition away from it: https:\/\/docs.rc.fas.harvard.edu\/kb\/centos7-singularity\/<\/a><\/p>\n If you load modules in your .bashrc<\/a> or other startup script, you will likely need to modify this as some modules will no longer be available or will have changed. This is why trying out the test cluster ahead of time is important. Software you have compiled may need to be re-compiled, as an example.<\/p>\n FASRC has a Rocky Linux test environment available where groups can begin to test their software, codes, jobs, etc.\u00a0 Please see the Rocky 8 Transition Guide<\/a> for details.<\/p>\n Regarding OOD\/VDI:<\/strong> The Open OnDemand instances for Cannon will involve a change to each user's settings and the location of the folder containing those settings and other OOD-specific files. This also means that values you had filled out in the OOD start form will not be retained. Form values will revert to the defaults.<\/p>\n The new location in your Cannon home directory will be ~\/.fasrcood (This does not affect FASSE OOD users which continues to use ~\/.fasseood)<\/p>\n Starting in April, FASRC will, in addition to our regular New User training and other upcoming classes<\/a>, will be offering training sessions on:<\/p>\n Additionally, we will be providing opportunities for labs and other groups to meet with us to discuss your workflows vis-a-vis the upgrade and changes. Please contact FASRC if your lab would like to schedule a consult.<\/p>\n DOWNTIME COMPLETE The annual multi-day power downtime at MGHPCC (https:\/\/www.rc.fas.harvard.edu\/blog\/2023-downtime\/) is complete (with exceptions noted below). Normal service resumes today (Friday June 9th) at 9am. As noted in previous communications, many changes have been made to the cluster and software. If you run jobs on the cluster\u00a0and did not previously try out the test cluster, you will need to make…<\/p>\ngpu_test<\/code>, fasse_gpu<\/code>, and remoteviz<\/code> partitions are not yet available due to unexpected configuration issues.\u00a0 It is our first priority to get them all online again as soon as we can. Other\u00a0GPU partitions are not affected.\u00a0<\/p>\n
\n- Nodes with GPUs older than V100 may also be down due to being dropped from support by current Nvidia drivers. We will need to address these on a case-by-case basis.<\/p>\nCHANGES<\/h2>\n
Operating System\u00a0<\/h3>\n
Software<\/h3>\n
\nPublic Partition Time Limits on Cannon<\/h3>\n
TRAINING\/SUPPORT<\/h3>\n
\nOffice Hours resumes Wednesday June 14th. Three additional special office hours have been added for 6\/15, 6\/22, 6\/29.
\nPlease see: https:\/\/www.rc.fas.harvard.edu\/training\/office-hours\/<\/a><\/p>\nIMPORTANT NOTES BEFORE LOGGING IN<\/h2>\n
Module Loads\/.bashrc<\/h3>\n
\nYou can also remove \"source new-modules.sh\" from your .bashrc if it is still there.<\/p>\nSSH Error:\u00a0<\/h3>\n
\n@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED @<\/code>
\nThis is because new login nodes will have been deployed but with the same names.
\nSee these instructions for removing the old key fingerprint(s): https:\/\/docs.rc.fas.harvard.edu\/kb\/ssh-key-error\/<\/a><\/p>\nConda\/Anaconda Python Environments<\/h3>\n
Login Node Processes - Core and Memory Restrictions<\/h3>\n
\nShould you run a process on a login node that runs afoul of cgroup policies, the process will be killed. Please be aware that\u00a0there is no warning mechanism. Please err on the side of caution when choosing to run something on a login node versus an\u00a0interactive session.
\nSee: https:\/\/docs.rc.fas.harvard.edu\/kb\/rocky-8-transition-guide\/<\/a> and\u00a0https:\/\/docs.rc.fas.harvard.edu\/kb\/responsibilities\/<\/a><\/p>\n\nOOD\/OpenOnDemand\/VDI<\/h3>\n
\nThe old folder at ~\/fasrc will no longer be used and can be removed.
\nThe location for FASSE settings will not change and will remain at ~\/.fasseood<\/p>\nPOST-UPGRADE SUPPORT<\/h3>\n
\nFAS Research Computing
\nhttps:\/\/www.rc.fas.harvard.edu\/<\/a>
\nhttps:\/\/docs.rc.fas.harvard.edu\/<\/a>
\nhttps:\/\/status.rc.fas.harvard.edu\/<\/a>
\nrchelp@rc.fas.harvard.edu\u00a0\u00a0<\/p>\n
\nOriginal Timeline and Information<\/h2>\n
![\"\"](\"https:\/\/www.rc.fas.harvard.edu\/wp-content\/uploads\/2023\/02\/mghpcc_powerdown_2023.jpg\")
<\/p>\nTest clusters for Cannon and FASSE (including OOD\/VDI) are available<\/a>.<\/del> Due to the breadth of these changes, this downtime will also impact almost every service we provide, including most housed in Boston.<\/p>\nWARNING: REMOTE HOST IDENTIFICATION HAS CHANGED<\/code>
\nThis is because new login nodes will have been deployed but with the same names. See these instructions for removing the old key fingerprint.<\/a><\/p>\nOS Update - Rocky Linux<\/h3>\n
![\"\"](\"https:\/\/www.rc.fas.harvard.edu\/wp-content\/uploads\/2023\/02\/rockylinux-300x99.jpg\")
<\/p>\nCannon and FASSE OOD (aka VDI)<\/h3>\n
Public Partition Time Limits on Cannon<\/h3>\n
Software Changes<\/h3>\n
![\"\"](\"https:\/\/www.rc.fas.harvard.edu\/wp-content\/uploads\/2023\/02\/spack-300x90.jpg\")
FASRC will reduce the number of precompiled software packages that it hosts, distilling this down to the necessities of compilers, any commercial packages, apps needed for VDI, etc.\u00a0<\/p>\nTraining, Office Hours, and Consultation<\/h3>\n
\n
\nLinks to this town hall meeting and its slides can be found at https:\/\/docs.rc.fas.harvard.edu\/kb\/rocky-8-transition-guide\/<\/li>\n\n
FASRC has a Rocky Linux 8 test environment available where groups can begin to test their software, codes, jobs, etc.\u00a0 This includes Cannon and FASSE users and OOD\/VDI.<\/del> Please see the Rocky 8 Transition Guide<\/a> for details.<\/p>\n","protected":false},"excerpt":{"rendered":"