{"id":25828,"date":"2025-10-10T11:22:31","date_gmt":"2025-10-10T15:22:31","guid":{"rendered":"https:\/\/www.rc.fas.harvard.edu\/?page_id=25828"},"modified":"2026-08-05T09:18:29","modified_gmt":"2026-08-05T13:18:29","slug":"data-storage","status":"publish","type":"page","link":"https:\/\/www.rc.fas.harvard.edu\/data-storage\/","title":{"rendered":"Research Computing Storage Modernization Initiative"},"content":{"rendered":"<h3><b>Improving Data Storage Infrastructure at FAS<\/b><\/h3>\n<p><span style=\"font-weight: 400;\">FAS Research Computing (FAS RC) will be transitioning to a new storage infrastructure. The project will consolidate and modernize a significant portion of the existing storage filesystems and move to new and improved hardware. The new storage<\/span><span style=\"font-weight: 400;\"> will replace outdated filesystems, realign storage options with data lifecycle needs, incorporate enterprise-grade features, and establish a scalable and financially sustainable service model.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">The project incorporates over <strong>70 pebibytes<\/strong> of new data storage to ensure FAS RC remains at the forefront of research and meet the needs of the Harvard community.\u00a0\u00a0<\/span><\/p>\n<h3><b>Strategic Outcomes<\/b><\/h3>\n<ul>\n<li><b>Sustainable Growth<\/b><b><br \/>\n<\/b><span style=\"font-weight: 400;\">Scalable, cost-effective storage aligned to support expanding research demands and lifecycle trends.<\/span><span style=\"font-weight: 400;\"><br \/>\n<\/span><\/li>\n<li><b>Improved Service Quality<\/b><b><br \/>\n<\/b><span style=\"font-weight: 400;\">Resilient infrastructure with enterprise-grade reliability and support creating a user-friendly experience.<\/span><span style=\"font-weight: 400;\"><br \/>\n<\/span><\/li>\n<li><b>Operational Efficiency<\/b><b><br \/>\n<\/b><span style=\"font-weight: 400;\">Reduces manual overhead, eliminates disruptive migrations, and reallocates staff to strategic and user-facing initiatives.<\/span><span style=\"font-weight: 400;\"><br \/>\n<\/span><\/li>\n<li><b>Financial Stability<\/b><b><br \/>\n<\/b><span style=\"font-weight: 400;\">Establishes a predictable long-term cost-recovery model with transparent tiered pricing and vendor-aligned incentives.<\/span><span style=\"font-weight: 400;\"><br \/>\n<\/span><\/li>\n<li><b>Forward-thinking Infrastructure<\/b><b><br \/>\n<\/b><span style=\"font-weight: 400;\">Supports AI\/ML, secure multi-protocol access, and evolving scientific workflows.<\/span><\/li>\n<\/ul>\n<h3><b>Storage Growth &amp; Future Demand<\/b><\/h3>\n<p><span style=\"font-weight: 400;\">FASRC storage capacity grew from <\/span><b>32.72 PiB to 87.89 PiB<\/b><span style=\"font-weight: 400;\"> (a <\/span><b>168.5% increase)<\/b><span style=\"font-weight: 400;\"> over the past several years, reflecting an average growth of <\/span><b>11.03 PiB per year<\/b><span style=\"font-weight: 400;\"> or <\/span><b>21%. <\/b>Storage demand is projected to reach <b>141 PiB by FY30.\u00a0<\/b><\/p>\n<p><span style=\"text-decoration: underline;\"><span style=\"font-weight: 400;\">Key growth drivers include:<\/span><\/span><span style=\"font-weight: 400;\"><br \/>\n<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">New lab and research group expansions<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Data retention requirements\u00a0<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Legacy data migrations<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">Improved data lifecycle enforcement and better management practices supported by analytics tools like <a href=\"https:\/\/docs.rc.fas.harvard.edu\/kb\/starfish-data-management\/\">Starfish<\/a> and <a href=\"https:\/\/docs.rc.fas.harvard.edu\/kb\/coldfront-allocation-management\/\">Coldfront<\/a> will help mitigate future growth, enabling users and administrators to make more informed decisions, optimize storage usage, and enhance internal forecasting.<\/span><\/p>\n<h2><b>New Storage Architecture<\/b><\/h2>\n<ul>\n<li>\n<h3><b>Compute Storage<\/b><\/h3>\n<ul>\n<li><b>20 PiB<\/b><span style=\"font-weight: 400;\"> of high-performance cluster adjacent storage<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Active storage for data analysis; designed to house data readily utilized and accessed.\u00a0<\/span><\/li>\n<li>Intended for HPC, AI\/ML, and parallel workloads<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<ul>\n<li>\n<h3><b>Lab Storage<\/b><\/h3>\n<ul>\n<li><strong>35 PiB<\/strong> of lab storage with <strong>backups (snapshots) and disaster recovery*<\/strong><\/li>\n<li><span style=\"font-weight: 400;\">General purpose storage for raw and project data; can be utilized as buffer storage for lab instruments and equipment<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Not intended for heavy computational workflows<\/span><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<ul>\n<li>\n<h3><b>FAS Secure Environment (FASSE)<\/b><\/h3>\n<ul>\n<li><strong>2.5 PiB<\/strong> secure storage with <strong>backups (snapshots) and disaster recovery*\u00a0<\/strong><\/li>\n<li><span style=\"font-weight: 400;\">Secure storage environment for sensitive or regulated data; data generated using Data Use Agreements (DUAs) or IRB<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Encryption at rest also included.<\/span><span style=\"font-weight: 400;\">\n<p><\/span><\/li>\n<\/ul>\n<\/li>\n<li>\n<h3><b>Long-Term Storage (Cold)**<\/b><\/h3>\n<ul>\n<li><span style=\"font-weight: 400;\"><strong>20 PiB<\/strong> of disk-based S3-compatible storage <strong>with disaster recovery***<\/strong><\/span><span style=\"font-weight: 400;\"><br \/>\n<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Long-term storage of research data to meet institutional data retention and compliance requirements. <\/span><\/li>\n<li><span style=\"font-weight: 400;\">On-premise long-term storage option for Harvard affiliated labs that is easier to access and faster than alternative long-term storage options (Tape)\u00a0<\/span><\/li>\n<\/ul>\n<\/li>\n<li>\n<h3><strong>Tape (NESE)\u00a0<\/strong><\/h3>\n<ul>\n<li>Existing long-term storage option <strong>without disaster recovery<\/strong><\/li>\n<li>Provided by the Northeast Storage Exchange (NESE) to house inactive research data after project completion or for data retention purposes.<\/li>\n<li>Stored on physical tapes in 20TB increments. Limited to ten thousand files per folder and file sizes between 1GiB to 100 GiB. You need to tar the data with small files to fulfill the technology requirement<\/li>\n<li><a href=\"https:\/\/docs.rc.fas.harvard.edu\/kb\/tape-globus-access\/\">Data transferred to and retrieved from Tape using the Globus tool.<\/a> \u00a0<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>*Snapshots are copies of a directory taken at a specific moment in time. They offer labs a self-service recovery option for overwritten or deleted files within the specific time period. Disaster recovery is a copy of an entire file system that can be used internally by FASRC in case of system-wide failure.<\/p>\n<p>** ECS object storage.\u00a0<\/p>\n<p>*** Disaster recovery is an additional cost.\u00a0<\/p>\n<h2>Storage Costs\u00a0<\/h2>\n<p>* New storage costs has been implemented for Tier 1 labs starting January 1, 2026. All Tier 1 storage allocations will be now be charged a rate of $125\/TB\/year.\u00a0<\/p>\n<table style=\"width: 40%; height: 285px; float: left;\">\n<tbody>\n<tr style=\"height: 27px;\">\n<td style=\"width: 116.566%; text-align: center; height: 27px;\" colspan=\"2\"><span style=\"font-size: 18px;\"><strong>Former Storage Tiers and Costs<\/strong><\/span><\/td>\n<\/tr>\n<tr style=\"height: 56px;\">\n<td style=\"width: 55.9406%; height: 56px;\">\n<b>Tier<\/b>\n<\/td>\n<td style=\"width: 60.6256%; height: 56px;\">\n<b>Rate ($\/TB\/year)<\/b>\n<\/td>\n<\/tr>\n<tr style=\"height: 80px;\">\n<td style=\"width: 55.9406%; height: 80px;\">\n<span style=\"font-weight: 400;\">Tier 0<\/span>\n<\/td>\n<td style=\"width: 60.6256%; height: 80px;\">\n<span style=\"font-weight: 400;\">$50<\/span>\n<\/td>\n<\/tr>\n<tr style=\"height: 10px;\">\n<td style=\"width: 55.9406%; height: 10px;\">\n<span style=\"font-weight: 400;\">Tier 1<\/span>\n<\/td>\n<td style=\"width: 60.6256%; height: 10px;\">\n<span style=\"font-weight: 400;\">$250<\/span>\n<\/td>\n<\/tr>\n<tr style=\"height: 56px;\">\n<td style=\"width: 55.9406%; height: 56px;\">\n<span style=\"font-weight: 400;\">Tier 2<\/span>\n<\/td>\n<td style=\"width: 60.6256%; height: 56px;\">\n<span style=\"font-weight: 400;\">$100<\/span>\n<\/td>\n<\/tr>\n<tr style=\"height: 56px;\">\n<td style=\"width: 55.9406%; height: 56px;\">\n<span style=\"font-weight: 400;\">Tier 3 (NESE Tape)<\/span>\n<\/td>\n<td style=\"width: 60.6256%; height: 56px;\">\n<span style=\"font-weight: 400;\">$5<\/span>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<table style=\"width: 20%; float: left; border: none;\" border=\"0\">\n<tbody style=\"border: none;\">\n<tr style=\"border: none;\">\n<td style=\"border: none;\">\u00a0<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<table style=\"width: 40%; height: 349px; float: left;\">\n<tbody>\n<tr>\n<td style=\"width: 331px; text-align: center;\" colspan=\"2\"><span style=\"font-size: 18px;\"><strong>New Storage Options and Costs<\/strong><\/span><\/td>\n<\/tr>\n<tr>\n<td style=\"width: 186px;\">\n<b>Tier<\/b>\n<\/td>\n<td style=\"width: 145px;\">\n<b>Rate ($\/TB\/year)<\/b>\n<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 186px;\">\n<span style=\"font-weight: 400;\">Compute Storage<\/span>\n<\/td>\n<td style=\"width: 145px;\">\n<span style=\"font-weight: 400;\">$150<\/span>\n<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 186px;\">\n<span style=\"font-weight: 400;\">Lab Storage <\/span>\n<\/td>\n<td style=\"width: 145px;\">\n<span style=\"font-weight: 400;\">$125<\/span>\n<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 186px;\">\n<span style=\"font-weight: 400;\">FASSE (Secure Enclave)<\/span>\n<\/td>\n<td style=\"width: 145px;\">\n<span style=\"font-weight: 400;\">$150<\/span>\n<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 186px;\">\n<span style=\"font-weight: 400;\">Long-Term Storage\u00a0<\/span>\n<\/td>\n<td style=\"width: 145px;\">\n<span style=\"font-weight: 400;\">$30<\/span>\n<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 186px;\">\n<span style=\"font-weight: 400;\">Tape (NESE)\u00a0<\/span>\n<\/td>\n<td style=\"width: 145px;\">\n<span style=\"font-weight: 400;\">$15<\/span>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h2>\u00a0<\/h2>\n<h2>Storage Transition Schedule\u00a0<\/h2>\n<p><span style=\"font-weight: 400;\">The migration to new storage hardware will occur in phases, beginning January 2026 through Spring next year. We will provide additional detailed guidance via email in advance of the migrations to ensure you are fully informed and prepared. We appreciate your patience and support through this transition.\u00a0<\/span><\/p>\n<table style=\"width: 63.0979%;\">\n<tbody>\n<tr>\n<td style=\"width: 30.3338%;\">\n<b>Current Storage<\/b>\n<\/td>\n<td style=\"width: 16.6909%;\">\n<b>Previous Tier<\/b>\n<\/td>\n<td style=\"width: 26.27%;\">\n<b>New Storage Offering<\/b>\n<\/td>\n<td style=\"width: 30.0435%;\">\n<b>Timeframe<\/b>\n<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 30.3338%;\">\n<span style=\"font-weight: 400;\">bos-isilon\/holy-isilon<\/span>\n<\/td>\n<td style=\"width: 16.6909%;\">\n<span style=\"font-weight: 400;\">Tier 1<\/span>\n<\/td>\n<td style=\"width: 26.27%;\">\n<span style=\"font-weight: 400;\">Lab Storage<\/span>\n<\/td>\n<td style=\"width: 30.0435%;\">\n<span style=\"font-weight: 400;\">Complete (August 3)<\/span>\n<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 30.3338%;\">\nboslfs02\n<\/td>\n<td style=\"width: 16.6909%;\">\n<span style=\"font-weight: 400;\">Tier 0<\/span>\n<\/td>\n<td style=\"width: 26.27%;\">\n<span style=\"font-weight: 400;\">Lab Storage<\/span>\n<\/td>\n<td style=\"width: 30.0435%;\">\n<span style=\"font-weight: 400;\">Phase 2 (mid-August)*<\/span>\n<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 30.3338%;\">\n<span style=\"font-weight: 400;\">FAS Secure Enclave (FASSE)<\/span>\n<\/td>\n<td style=\"width: 16.6909%;\">\n<span style=\"font-weight: 400;\">Various<\/span>\n<\/td>\n<td style=\"width: 26.27%;\">\n<span style=\"font-weight: 400;\">FASSE<\/span>\n<\/td>\n<td style=\"width: 30.0435%;\">\n<span style=\"font-weight: 400;\">Phase 3 (late August)*<\/span>\n<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 30.3338%;\">\n<span style=\"font-weight: 400;\">b-nfs-01\/b-nfs-10<\/span>\n<\/td>\n<td style=\"width: 16.6909%;\">\n<span style=\"font-weight: 400;\">Tier 2<\/span>\n<\/td>\n<td style=\"width: 26.27%;\">\n<span style=\"font-weight: 400;\">Lab Storage<\/span>\n<\/td>\n<td style=\"width: 30.0435%;\">\n<span style=\"font-weight: 400;\">Phase 4 (September)*<\/span>\n<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 30.3338%;\">\n<span style=\"font-weight: 400;\">h-nfs15\/h-nfs-20<\/span>\n<\/td>\n<td style=\"width: 16.6909%;\">\n<span style=\"font-weight: 400;\">Tier 2<\/span>\n<\/td>\n<td style=\"width: 26.27%;\">\n<span style=\"font-weight: 400;\">Lab Storage<\/span>\n<\/td>\n<td style=\"width: 30.0435%;\">\n<span style=\"font-weight: 400;\">Phase 4 (September)*<\/span>\n<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 30.3338%;\">\nholystore01\n<\/td>\n<td style=\"width: 16.6909%;\">\n<span style=\"font-weight: 400;\">Tier 0<\/span>\n<\/td>\n<td style=\"width: 26.27%;\">\n<span style=\"font-weight: 400;\">Compute Storage<\/span>\n<\/td>\n<td style=\"width: 30.0435%;\">\n<span style=\"font-weight: 400;\">Phase 5 (October)*<\/span>\n<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 30.3338%;\">\nholylfs05\/holylfs06 (Lustre)\n<\/td>\n<td style=\"width: 16.6909%;\">\n<span style=\"font-weight: 400;\">Tier 0<\/span>\n<\/td>\n<td style=\"width: 26.27%;\">\n<span style=\"font-weight: 400;\">Compute Storage<\/span>\n<\/td>\n<td style=\"width: 30.0435%;\">\n<span style=\"font-weight: 400;\">Phase 5 (October)*<\/span>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p><span style=\"font-weight: 400;\">*Specific timelines and dates will be communicated directly to affected labs prior to the migration. Netscratch, \/holylabs, and home folders will not be affected by this migration.\u00a0<\/span><\/p>\n<h2>Contact\u00a0<\/h2>\n<p><span style=\"font-weight: 400;\">If you have any questions regarding the new storage environment, please email the <a href=\"mailto:rchelp@rc.fas.harvard.edu\">FASRC Storage Migration team<\/a> and a member of FASRC will respond within two business days. <\/span><\/p>\n<p><span style=\"font-weight: 400;\">Please designate an individual within the lab or group that can act as a <a href=\"https:\/\/docs.rc.fas.harvard.edu\/kb\/roles-responsibilities\/\">General Manager<\/a>, or data manager to help track storage usage and communicate with the <a href=\"https:\/\/www.rc.fas.harvard.edu\/about\/people\/sarah-marchese\/\">FASRC Research Data Manager.<\/a> General managers also have the ability to review and monitor the group's storage usage using data management tools such as <a href=\"https:\/\/docs.rc.fas.harvard.edu\/kb\/starfish-data-management\/\">Starfish<\/a> and <a href=\"https:\/\/docs.rc.fas.harvard.edu\/kb\/coldfront-allocation-management\/\">Coldfront<\/a>. If you have any questions or would like to discuss data cleanup efforts, please email the FASRC Research Data Manager at <a href=\"mailto:rdm@rc.fas.harvard.edu\">rdm@rc.fas.harvard.edu<\/a>.\u00a0<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p class=\"lead\">Improving Data Storage Infrastructure at FAS FAS Research Computing (FAS RC) will be transitioning to a new storage infrastructure. The project will consolidate and modernize a significant portion of the existing storage filesystems and move to new and improved hardware. The new storage will replace outdated filesystems, realign storage options with data lifecycle needs, incorporate enterprise-grade features, and establish a&hellip;<\/p>\n<p class=\"more-link-p\"><a class=\"btn btn-primary\" href=\"https:\/\/www.rc.fas.harvard.edu\/data-storage\/\">Read more<\/a><\/p>\n","protected":false},"author":103,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"categories":[],"tags":[],"class_list":["post-25828","page","type-page","status-publish","hentry"],"jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/P42YvN-6IA","_links":{"self":[{"href":"https:\/\/www.rc.fas.harvard.edu\/wp-json\/wp\/v2\/pages\/25828","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.rc.fas.harvard.edu\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.rc.fas.harvard.edu\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.rc.fas.harvard.edu\/wp-json\/wp\/v2\/users\/103"}],"replies":[{"embeddable":true,"href":"https:\/\/www.rc.fas.harvard.edu\/wp-json\/wp\/v2\/comments?post=25828"}],"version-history":[{"count":42,"href":"https:\/\/www.rc.fas.harvard.edu\/wp-json\/wp\/v2\/pages\/25828\/revisions"}],"predecessor-version":[{"id":26058,"href":"https:\/\/www.rc.fas.harvard.edu\/wp-json\/wp\/v2\/pages\/25828\/revisions\/26058"}],"wp:attachment":[{"href":"https:\/\/www.rc.fas.harvard.edu\/wp-json\/wp\/v2\/media?parent=25828"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.rc.fas.harvard.edu\/wp-json\/wp\/v2\/categories?post=25828"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.rc.fas.harvard.edu\/wp-json\/wp\/v2\/tags?post=25828"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}