Starfish Zones is a self-service data management visualization tool that allows groups to view folder storage amounts and locations. Group members can navigate through the folder structures in the dashboard to explore directory and file level details, including storage amounts, last accessed and modified times, file owners, and file counts. The tool is still under development and may experience short downtimes to accommodate modifications.

Login

To access the dashboard, navigate to https://starfish.rc.fas.harvard.edu/. You will need to be on Harvard VPN, FASRC VPN, or a wired on-campus connection. See VPN Setup for how to connect to FASRC VPN.

After navigating to the website, input your FASRC account name and password. If you have difficulties with your password, you can reset it. You may also need to clear the cache on your website browser (Chrome and Firefox preferred).

If you are unable to access the Starfish dashboard, please email FASRC Help. If you are a new faculty or group owner, it may take some time before your information is fully populated in FASRC systems. You will also need an active FASRC supported storage folder to receive a Starfish Zone.

Navigation

Once logged in, you will be able to view storage folders associated with your group. Note: If you notice a storage folder is missing from your dashboard, please email FASRC’s Research Data Manager. A majority of storage folders supported by FASRC are now viewable in Starfish, some filesystems still need to be added.

Overview

The Overview view offers a summary of the Zone, including total size and count (number of files and directories). This is helpful for understanding all storage folders assigned to your group.

Sunburst

The Sunburst view provides visualizations of subdirectories within the Zone based on desired characteristics, such as last modification date or username. The image below demonstrates how the Sunburst visualization can assist with data cleanup efforts by showcasing subdirectories that have not been modified for several years. These folders could be ideal candidates for data review and cleanup.

Browse

One of the more useful views will be the Browse view. By double-clicking on a selected folder path, you can drill down into the folder all the way to the file level. Group members can modify what information is displayed on the dashboard by right-clicking on the column headers. All available column options will be shown.

The dashboard is updated on a consistent basis. You can view when the Zone was last updated on the lower right-hand corner of the dashboard, where a date and time will be listed. If no modifications have been made to the folder contents, the updated time will reflect the last time changes were made to the folder.

To view a summary of the directory, hover over the badge next to the directory name to see a pop-up with more detailed statistics, including size and count. This allows you to quickly see key information about your current Zone without needing to open the additional details panel.

History

The History view allows group members to select a predefined timeframe to investigate further, using the calendar to choose two custom dates. The interface will highlight the status of each item, to help identify elements that have been created, removed, or changed in size between the two dates.

Export

The dashboard also allows group members to export the information as a CSV file. At the top of the zone is a “CSV” dropdown option.

Depending on the number of files and folders listed, the CSV file may be too large to download. We recommend selecting the specific subfolder that you would like to view as an export, to decrease the size of the downloaded file.

Tarring Data for Tape with Starfish Tags

Starfish can also be used to tar the contents of a folder in preparation for migration to NESE Tape storage. The tarring script is designed to tar files “in place,” tarring files in the source folder to prevent issues with storage capacity.

To have this tagging option enabled for your lab, please email rdm@rc.fas.harvard.edu, which will initiate a ticket. We will then enable the ‘tag’ for your Starfish dashboard.

Once the tag has been enabled, you can follow the steps below to initiate the process on your own. Keep in mind that Starfish’s script creates tar files that are compliant with NESE Tape requirements. For example, a Tape allocation of 200 TB will have a default limit on the number of files set to 2,000,000 (200 TB / 100 MB).

Applying the Tag

• Navigate to the folder in the Starfish interface that you want to tar. The tag only needs to be
  applied once for a given directory tree; everything under the tag will be tarred so ensure you are selecting the correct level.
• Right click on the folder and select “Classification Tags” and apply the tag
  “fasrc_tagset:tar-in-place”
• The automated script will run hourly once the tag is applied (on the half hour).
• The process will show as completed when ‘manifest files’ with the following naming convention appear in the parent folder, where the tag was applied: “manifest_1234_tar_wrapper_YYYYMMDD_csv.gz”. The information and date in the naming convention will change depending on when the tag was applied and the folder contents.

Tag Workflow

• The script is designed to prepare files for migration to Tape storage, with those specific size constraints in mind. The files will be compressed to approximately 1GB in size.
• The script will continue to tar any uncompressed files in the folder. If new files or folders are added to a tagged folder, the script will begin again to compress the files. The script is configured to run hourly, selecting which step to start on depending on the current state of the tagged folder.
• Once all files are compressed and the tarring jobs have completed, the original ownership permissions will be reapplied and the manifest files (listed above) will be generated. This may take a day or two to be applied, as it relies on the latest Starfish scan. We recommend keeping the tag on the directory for a day or two to ensure the permissions are applied.
• When the files are tarred, removed, manifest file created, and permissions applied, you can remove the tag. The script will continue to run on an hourly basis until the tag is removed.

Contact

If you have any additional questions about how to login or utilize the Starfish Zones dashboard, please email Sarah Marchese, FASRC Research Data Manager.

Accessing Coldfront

To access Coldfront, connect to the FAS RC VPN and log in using your FASRC username and password. If you have difficulties with your password, you can reset it. You may also need to clear the cache on your website browser.

Homepage

After logging in to the dashboard, you will see a home page, with sections for your projects, storage allocations, and any pending requests or change requests for your allocations.

Click the Projects link or Allocations option to view details about a given project or allocation.

Project Pages

The Project page allows all project members to:

• view the project’s storage allocations
• view the project’s group members
• adjust project notifications

General Managers, Storage Managers, and PIs can also (see below for further information about roles/responsibilities):

• request new storage allocations
• request changes to existing storage allocations

General Managers, Access Managers, and PIs can also:

• add members to the project
• remove members from the project

PIs and General Managers can:

• edit the roles of project members (i.e., assign or remove Access or Storage Manager status)

Project Allocations

The Project Allocations page provides a comprehensive view of the group’s storage allocations, showcasing detailed information including total allocation size, overall usage, and estimated monthly cost.

To view further information about the storage allocation, select View Details. The new page will display detailed information about the storage allocation, including a list of group members with data stored in the directory. Data housed in the storage allocation will be displayed in the table regardless of the active status of the original owner. If a dataset was generated by a lab member who has left the lab or university, but the data still resides in the lab folder, the individual will be listed in the member list, as they are the original owner.

Project Membership and Roles

PIs, Access Managers, and General Managers can see the option to add and remove members from the project in the group membership table. PIs and General Managers can additionally edit the roles of project members and send emails to project members that have elected to receive notifications.

The Project roles correspond to the roles as described in FASRC Roles and Responsibilities:

• PI. This role is automatically assigned to the group’s sponsor and/or PI and cannot be assigned to another Project member. The PI can:
  • Request new storage allocations
  • Request changes to existing storage allocations
  • Add and/or remove project members
  • Change the roles of existing project members
• General Manager. General Managers are assigned by PIs. They have all the permissions of a PI, save the ability to assign a General Manager role to other project members.
• Storage Manager. Storage Managers are assigned by PIs and General Managers. Storage Managers can:
  • Request new storage allocations
  • Request changes to existing storage allocations
• Access Manager. Access Managers are assigned by PIs and General Managers. Access Managers can:
  • Add and/or remove project members
• User. Project members can view project and allocation information. They cannot modify the project membership or storage allocations.

Requesting a New Storage Allocation:

PIs, General Managers, and Storage Managers can request a new storage allocation or make changes to existing project allocations in Coldfront.

To request a new allocation:

1. Go to the Project Allocations section and select Request New Storage Allocation.
2. Fill out and submit the allocation request form. You may need a 33-digit billing code to submit the request.
3. You will be notified via email when your new allocation is ready to use. It may take up to a week for the allocations to be created.

Allocations can be requested on several of the FAS RC storage offerings. To explore and understand the specifics of each storage option, please refer to our Data Storage Workflows documentation.

Submitting an Allocation Change Request:

PIs, General Managers, and Storage Managers can request a change to the size of a storage allocation associated with their project. Follow these steps to initiate the process:

1. Navigate to the Project Allocations section and the allocation you wish to modify.
2. Select the Request Allocation Change option.
3. In the resulting form, shown below, enter the total desired size of the allocation (TiBs) and the justification for the changes and click submit.
   • For example, if your lab has a 15TiB storage allocation and you are requesting an additional 5TiB, you would input 20TiB as the requested new value.
4. You will be notified via email when the allocation is updated and ready to use. Space permitting, we try to fulfill change requests within 3-5 business days.

Allocation History

The Allocation History section showcases all of the storage allocations that have been associated with the group over time, including storage allocations that have been deactivated. It also lists the dates when the allocations were created or deactivated.

Group Membership

The Users section features a table illustrating usage per group member, with data sourced and updated daily from our data visualization tool, Starfish.

PIs, General Managers, and Access Managers can add or remove group members using the provided buttons.

PIs and General Managers can also modify the role of the individual by clicking Actions and selecting the desired role based on the available options in the dropdown menu. 

For more information about the available storage offerings, including their features and costs, please review the Data Storage Workflow documentation on our website.

If you have any questions, please email us at rchelp@rc.fas.harvard.edu.