{"id":25500,"date":"2024-11-03T15:04:04","date_gmt":"2024-11-03T20:04:04","guid":{"rendered":"https:\/\/www.rc.fas.harvard.edu\/?p=25500"},"modified":"2024-12-12T10:06:12","modified_gmt":"2024-12-12T15:06:12","slug":"announcing-netscratch","status":"publish","type":"post","link":"https:\/\/www.rc.fas.harvard.edu\/blog\/announcing-netscratch\/","title":{"rendered":"Announcing Netscratch: Our New High-Performance Scratch Filesystem\u00a0"},"content":{"rendered":"<p><em>This was sent as an email(s) to all cluster users on Nov 1st, 2024. It has been edited here to provide additional information about moving data.<\/em><\/p>\n<p>We\u2019re excited to introduce\u00a0<strong>netscratch<\/strong>, a high-performance scratch filesystem powered by\u00a0<strong>Vast Data<\/strong>. After thorough evaluation, Vast Data\u2019s advanced flash technology stood out for its performance and scalability, making it our solution of choice.<\/p>\n<p>&nbsp;<\/p>\n<h3>Data Transfer from Holyscratch01 to Netscratch<\/h3>\n<p>Please copy\/move any needed data to\u00a0<code>\/n\/netscratch\/[your lab name]<\/code> (example: <em>\/n\/netscratch\/jharvard_lab<\/em> ) as soon as possible and begin using netscratch for your jobs. Not only do we want to encourage rapid adoption of netscratch, but we are seeing signs of degradation on holyscratch01 that increases the sense of urgency.<\/p>\n<p>See our <a href=\"https:\/\/docs.rc.fas.harvard.edu\/kb\/transferring-data-on-the-cluster\/\">Data Transfer documentation<\/a> for info on moving data. All labs who have directories on holyscratch01 should have directories on netscratch. If your lab does not, please <a href=\"https:\/\/www.rc.fas.harvard.edu\/about\/contact\/\">contact FASRC<\/a>.<\/p>\n<p><span style=\"color: #993300;\"><strong>Holy scratch01 will be made read-only on December 2nd, 2024. It will be decommissioned on Feb. 1st 2025.<\/strong><\/span><\/p>\n<p>&nbsp;<\/p>\n<h3>Key Benefits<\/h3>\n<p>The\u00a0<strong>netscratch<\/strong>\u00a0system leverages\u00a0<strong>NVMe<\/strong>\u00a0and\u00a0<strong>SSD<\/strong>\u00a0technol<wbr \/>ogies, offering:<\/p>\n<ul>\n<li><strong>Read Throughput:<\/strong>\u00a0Up to 250 GB\/s<\/li>\n<li><strong>Write Throughput:<\/strong>\u00a0Up to 55 GB\/s<\/li>\n<li><strong>IOPS:<\/strong>\u00a01,220,000 reads and 500,000 writes This system is optimized for the\u00a0<strong>Cannon cluster<\/strong>, allowing efficient job execution that maximizes both\u00a0<strong>CPU and GPU<\/strong>\u00a0resources.<\/li>\n<\/ul>\n<h3><a href=\"https:\/\/docs.rc.fas.harvard.edu\/kb\/data-storage-workflow-rdm\/\" target=\"_blank\" rel=\"noopener\" data-saferedirecturl=\"https:\/\/www.google.com\/url?q=https:\/\/docs.rc.fas.harvard.edu\/kb\/data-storage-workflow-rdm\/&amp;source=gmail&amp;ust=1730750358119000&amp;usg=AOvVaw1-6hIRqcauIqFQ59nmrcej\">Directory Structure for Enhanced Usability \u21b5<\/a><\/h3>\n<p><strong>netscratch<\/strong>\u00a0features a streamlined directory structure that will enhance usability<\/p>\n<ul>\n<li>Each lab will have a\u00a0<strong>top-level directory<\/strong>.<\/li>\n<li>Subdirectories for\u00a0<strong>\u201cEveryone\u201d<\/strong>\u00a0and\u00a0<strong>\u201cLab\u201d<\/strong>\u00a0will be included for common and lab-specific data. The \"<strong>Users<\/strong>\" directory is deprecated,\u00a0<a href=\"https:\/\/docs.rc.fas.harvard.edu\/kb\/data-storage-workflow-rdm\/#Default_Directory_Structure\" target=\"_blank\" rel=\"noopener\" data-saferedirecturl=\"https:\/\/www.google.com\/url?q=https:\/\/docs.rc.fas.harvard.edu\/kb\/data-storage-workflow-rdm\/%23Default_Directory_Structure&amp;source=gmail&amp;ust=1730750358119000&amp;usg=AOvVaw0th8QoHvTfIuhUv5kS_u6U\">please see this doc for info<\/a>.<\/li>\n<\/ul>\n<h3><a href=\"https:\/\/docs.rc.fas.harvard.edu\/kb\/policy-scratch\/\" target=\"_blank\" rel=\"noopener\" data-saferedirecturl=\"https:\/\/www.google.com\/url?q=https:\/\/docs.rc.fas.harvard.edu\/kb\/policy-scratch\/&amp;source=gmail&amp;ust=1730750358119000&amp;usg=AOvVaw3pjXU1GmvuILCoI8uVttns\">Technical Details \u21b5<\/a><\/h3>\n<ul>\n<li><strong>Filesystem Path<\/strong>:\u00a0<code>\/n\/netscratch<\/code>\u00a0(<wbr \/>accessible on Cannon and FASSE).<\/li>\n<li><strong>Quotas and Data Retention<\/strong>: Remain unchanged.<\/li>\n<li><strong>$SCRATCH Variable Update<\/strong>: Will point to\u00a0<strong>netscratch<\/strong>\u00a0starting December 2nd 2024.<\/li>\n<li><strong>External Repos Support<\/strong>: Users of\u00a0<code>\/n\/holyscratch01\/external_<wbr \/>repos<\/code>\u00a0should migrate their data as external repos will no longer be supported.<\/li>\n<\/ul>\n<h3>Launch Timeline<\/h3>\n<ul>\n<li><strong>Availability<\/strong>: Netscratch goes live on\u00a0<strong>Monday, November 4th 2024<\/strong>\u00a0after maintenance.<\/li>\n<li><strong>Transition Period<\/strong>:\n<ul>\n<li><strong>Write Access<\/strong>\u00a0on the current scratch filesystem will continue until\u00a0<strong>December 2nd 2024<\/strong>.<\/li>\n<li>After that, it will switch to\u00a0<strong>read-only<\/strong>\u00a0mode until\u00a0<strong>February 1st, 2025<\/strong>\u00a0(data retention policies remain active). We encourage users to start utilizing\u00a0<strong>netscratch<\/strong>\u00a0for upcoming jobs and to begin migrating data from the old scratch filesystem. For more on data transfers, please refer to the\u00a0<a href=\"https:\/\/docs.rc.fas.harvard.edu\/kb\/transferring-data-on-the-cluster\/\" target=\"_blank\" rel=\"noopener\" data-saferedirecturl=\"https:\/\/www.google.com\/url?q=https:\/\/docs.rc.fas.harvard.edu\/kb\/transferring-data-on-the-cluster\/&amp;source=gmail&amp;ust=1730750358119000&amp;usg=AOvVaw3E73WGmUCq9RDBUl9VGZGb\">the data transfer documentation<\/a>.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p>Thank you,<br \/>\nFAS Research Computing<br \/>\n<a href=\"https:\/\/docs.rc.fas.harvard.edu\/\" target=\"_blank\" rel=\"noopener\" data-saferedirecturl=\"https:\/\/www.google.com\/url?q=https:\/\/docs.rc.fas.harvard.edu\/&amp;source=gmail&amp;ust=1730750358119000&amp;usg=AOvVaw04vA2bvxAcuMc_mYFbkDq1\">https:\/\/docs.rc.fas.harvard.<wbr \/>edu\/<\/a><br \/>\n<a href=\"https:\/\/www.rc.fas.harvard.edu\/upcoming-training\/\" target=\"_blank\" rel=\"noopener\" data-saferedirecturl=\"https:\/\/www.google.com\/url?q=https:\/\/www.rc.fas.harvard.edu\/upcoming-training\/&amp;source=gmail&amp;ust=1730750358119000&amp;usg=AOvVaw0BcPdgIEHi8jjjGrroWRWY\">https:\/\/www.rc.fas.harvard.<wbr \/>edu\/upcoming-training\/<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p class=\"lead\">This was sent as an email(s) to all cluster users on Nov 1st, 2024. It has been edited here to provide additional information about moving data. We\u2019re excited to introduce\u00a0netscratch, a high-performance scratch filesystem powered by\u00a0Vast Data. After thorough evaluation, Vast Data\u2019s advanced flash technology stood out for its performance and scalability, making it our solution of choice. &nbsp; Data&hellip;<\/p>\n<p class=\"more-link-p\"><a class=\"btn btn-primary\" href=\"https:\/\/www.rc.fas.harvard.edu\/blog\/announcing-netscratch\/\">Read more<\/a><\/p>\n","protected":false},"author":103,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[36,170,198],"tags":[],"class_list":["post-25500","post","type-post","status-publish","format-standard","hentry","category-blog","category-cannon","category-fasse"],"jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p42YvN-6Di","jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/www.rc.fas.harvard.edu\/wp-json\/wp\/v2\/posts\/25500","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.rc.fas.harvard.edu\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.rc.fas.harvard.edu\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.rc.fas.harvard.edu\/wp-json\/wp\/v2\/users\/103"}],"replies":[{"embeddable":true,"href":"https:\/\/www.rc.fas.harvard.edu\/wp-json\/wp\/v2\/comments?post=25500"}],"version-history":[{"count":5,"href":"https:\/\/www.rc.fas.harvard.edu\/wp-json\/wp\/v2\/posts\/25500\/revisions"}],"predecessor-version":[{"id":25538,"href":"https:\/\/www.rc.fas.harvard.edu\/wp-json\/wp\/v2\/posts\/25500\/revisions\/25538"}],"wp:attachment":[{"href":"https:\/\/www.rc.fas.harvard.edu\/wp-json\/wp\/v2\/media?parent=25500"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.rc.fas.harvard.edu\/wp-json\/wp\/v2\/categories?post=25500"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.rc.fas.harvard.edu\/wp-json\/wp\/v2\/tags?post=25500"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}