This article provides a description on how to use Aspera On Demand and AWS life cycle policies to archive content to Glacier.
1) What is the difference between S3 and Glacier?
Answer: S3 is an object storage based storage service. Glacier is an archive service. With S3, you are able to access your content at any time. With Glacier, you need to request access to your content, and wait a few hours (typically 3-4 hours) before you can download it. Glacier costs less than S3. Additional details can be found on the AWS Glacier FAQ.
2) Can Aspera be used to move content to Glacier?
Answer: Yes. Indirectly via S3. S3 has a concept called 'Lifecycle rules' where you can assign a policy to migrate data from S3 to Glacier. One of the options is to migrate all data immediately. You can read more about this S3 feature in the S3 Documentation.
3) What are the benefits of using S3 to migrate content to Glacier?
Answer: There are a few. One is the lifecycle rules themselves. With these policies, you can migrate data to Glacier and eventually, expire data, when it is no longer needed. The other benefit is that all of the metadata about the object persists in S3, so you can do searches in your bucket to find files.
4) How can we setup Aspera to move content to Glacier?
Answer: Please see Setup details below.
Note: Since we leverage the S3 life cycle rules, setting up your Aspera On demand to move content to Glacier, first requires setting up your Aspera On Demand system for S3, and then setting up an S3 life cycle policy.
1) Setup your Aspera On Demand so that you can upload content to S3. See instructions for details.
2) Configure your S3 bucket with a life cycle policy to archive data to Glacier.
In my example, I have created a life cycle policy with two rules: Rule #1: migrate all files older than April 6 to Glacier. Rule # 2: Delete all files after May 2. The screenshots provided below, show the basic setup.
Screenshot #1: The bucket "sedemo-oregon-move-to-glacier" has a life cycle rule.
Screenshot #2: The details of the life cycle rule.
Screenshot #3: The details objects in the bucket that have been migrated to Glacier. NOTE: Storage class = Glacier.
The Aspera applications are designed to have a basic level of awareness of files are in Glacier. As you can see from the screenshots, the GUI application and the Shares application are both able to identify the files in Glacier.
Screenshot #4: Using Shares to browse content in Glacier. NOTE: The file icons indicate that the files are special.
Screenshot #5: Using Aspera Client GUI to browse content in Glacier. NOTE: The file icons indicate that the files are special.
Screenshot #6: Error message when trying to transfer a file from Glacier. Message: "Error - Not a regular file"