{"id":8153,"date":"2013-07-16T13:12:01","date_gmt":"2013-07-16T17:12:01","guid":{"rendered":"https:\/\/rcwebsite2.rc.fas.harvard.edu\/?p=8153"},"modified":"2017-12-05T10:46:21","modified_gmt":"2017-12-05T15:46:21","slug":"of-benching-a-wicked-fast-storage-array-with-a-very-dodgy-tcsh-script","status":"publish","type":"post","link":"https:\/\/www.rc.fas.harvard.edu\/blog\/of-benching-a-wicked-fast-storage-array-with-a-very-dodgy-tcsh-script\/","title":{"rendered":"Of benching a wicked fast storage array with a very dodgy tcsh script…"},"content":{"rendered":"

Testing our new secret filesystem... no caching or silly business going on here, all raw spindles and extremely suspect but reproducible fork bombs courtesy of tcsh.<\/p>\n

Where hosts contains 20 machines...<\/p>\n
[root@localhost<\/a>\u00a0jcuff]# foreach h (`cat hosts`)
\nforeach? ssh $h \/mnt\/share\/jcuff\/dd.sh $h &
\nforeach? end<\/div>\n

And where dd.sh is simply a 16 way fork bomb:<\/p>\n

[root@localhost<\/a>\u00a0jcuff]# cat dd.sh
\n#!\/bin\/tcsh
\nforeach h (`seq 1 16`)
\ndd if=\/dev\/zero of=\/mnt\/jcuff.$h.$1 bs=1024k count=64000 >& \/mnt\/jcuff.$h.$1.out &
\nend<\/div>\n

Which makes for\u00a0THREE HUNDRED AND TWENTY<\/strong>\u00a0concurrent jobs each writing out 64GB, or in layman's terms this is basically\u00a0TWENTY TERABYTES<\/strong>\u00a0of output.<\/p>\n

How would your current network attached storage hold up under this type of crazy load?<\/p>\n

Could it?<\/p>\n

Oh and I should say that this file system has another slightly unique property:<\/p>\n

\n
[root@localhost ~]# df -H\r\nFilesystem             Size   Used  Avail Use% Mounted on\r\n10.10.187.101:\/export\r\n                       1.2P   1.7T   1.1P   1% \/mnt\/<\/pre>\n<\/div>\n
This one seemed to, in only 25 minutes we had generated our 20 Terabytes!<\/p>\n
[root@compute001]# du -sh .
\n20T<\/strong><\/div>\n
As an aggregate speed we saw:<\/p>\n
\n
[root@compute001]# cat *.out | grep bytes \\\r\n| awk '{print (gbpersecond=gbpersecond+$8)\/1024 \" GB\/s\"}' \\\r\n| tail -1\r\n\r\n14.1553 GB\/s<\/strong><\/pre>\n<\/div>\n
we also confirmed speeds with a more fancy mpi run (still powered by tcsh ;-)):<\/p>\n
\n
[root@compute001 jcuff]# cat mpi.sh\r\n#!\/bin\/tcsh\r\nforeach h (`cat hosts`)\r\nssh $h \"\/usr\/bin\/mpirun -host $h -np 36\r\n\/mnt\/share\/openmpi_IOR\/src\/C\/IOR -b 2g -t 1m -F -C -w -k -e -vv -o\r\n\/mnt\/file1_p36_3.$h >&  \/mnt\/file1_p36_3.$h.out\" &\r\nend<\/pre>\n<\/div>\n
Needs a whole lot more testing, with like real science and stuff but as a start out with a three line dodgy tcsh script, this looks to be a mighty fine file system ;-) This is a teaser, I'm not giving out makes and models or any of that, just wanted to let folks know we are building an awesome monster here! Oh and also to help see if folks will ever stop making fun of me using tcsh. Hehehe!<\/p>\n
Summary: 320 jobs, 20 hosts, 20TB output @ 14GB\/s to 1.2PB<\/p><\/div>\n","protected":false},"excerpt":{"rendered":"
Testing our new secret filesystem... no caching or silly business going on here, all raw spindles and extremely suspect but reproducible fork bombs courtesy of tcsh. Where hosts contains 20 machines... [root@localhost\u00a0jcuff]# foreach h (`cat hosts`) foreach? ssh $h \/mnt\/share\/jcuff\/dd.sh $h & foreach? end And where dd.sh is simply a 16 way fork bomb: [root@localhost\u00a0jcuff]# cat dd.sh #!\/bin\/tcsh foreach h…<\/p>\n
Read more<\/a><\/p>\n","protected":false},"author":103,"featured_media":8913,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[36],"tags":[],"class_list":["post-8153","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog"],"jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p42YvN-27v","jetpack_featured_media_url":"https:\/\/www.rc.fas.harvard.edu\/wp-content\/uploads\/2013\/11\/1385493332_megafone.png","_links":{"self":[{"href":"https:\/\/www.rc.fas.harvard.edu\/wp-json\/wp\/v2\/posts\/8153","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.rc.fas.harvard.edu\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.rc.fas.harvard.edu\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.rc.fas.harvard.edu\/wp-json\/wp\/v2\/users\/103"}],"replies":[{"embeddable":true,"href":"https:\/\/www.rc.fas.harvard.edu\/wp-json\/wp\/v2\/comments?post=8153"}],"version-history":[{"count":1,"href":"https:\/\/www.rc.fas.harvard.edu\/wp-json\/wp\/v2\/posts\/8153\/revisions"}],"predecessor-version":[{"id":8154,"href":"https:\/\/www.rc.fas.harvard.edu\/wp-json\/wp\/v2\/posts\/8153\/revisions\/8154"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.rc.fas.harvard.edu\/wp-json\/wp\/v2\/media\/8913"}],"wp:attachment":[{"href":"https:\/\/www.rc.fas.harvard.edu\/wp-json\/wp\/v2\/media?parent=8153"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.rc.fas.harvard.edu\/wp-json\/wp\/v2\/categories?post=8153"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.rc.fas.harvard.edu\/wp-json\/wp\/v2\/tags?post=8153"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}