File time stamp preservation with Object Storage

Summary

This article describes how to configure Aspera to preserve file time stamps (creation time, access time, and modification time) for files that are being transferred via object storage (e.g. S3).  Aspera has support for the preservation of time stamps of files that are moved with Aspera.  We are now supporting that functionality on object storage platforms.   The design of this feature enables you to upload to and then download from Object Storage with the ability to preserve the original time stamps on the files and folders.  (folder create time preservation can be useful when using Shares to browse S3.)

Caveat:  This new option is being built into the 3.5 server version.   You will need to upgrade to that version in order to use this feature.

 

 

Details

Object storage platforms, do not have native support for preserving the modification, access or create times of files that have been moved to the object storage from traditional file systems.  In order to provide this functionality, Aspera designed a system for capturing the required time stamps as metadata attributes of the objects. This system was designed for general metadata capture, so in the future (once it is fully released) you will be able to associated many metadata tags to your objects.

 

Server Configuration

The new feature is enabled by a setting in aspera.conf.  This command, when run as root, can be used to turn on the feature:

# asconfigurator -x "set_node_data;preserve_attributes,times"
# asconfigurator -x "set_node_data;preserve_attributes_pvcl_read,enable"
# asconfigurator -x "set_node_data;preserve_attributes_pvcl_write,enable"

You can verify the setting, by viewing the contents in aspera.conf.  You should see a new entry in the 'file_system' section.  Here is an abbreviated example of what you will see:

 <default>
<file_system>
<preserve_attributes_pvcl_read>enable</preserve_attributes_pvcl_read>
<preserve_attributes_pvcl_write>enable</preserve_attributes_pvcl_write>
<preserve_attributes>times</preserve_attributes>
</file_system>
</default>

NOTE: I have removed many of the other default settings, for clarity.  

 

Client transfer Options

The Aspera client supports multiple options for preserving file time stamps.  The following 'ascp' options are described below

-p 
--preserve-modification-time
--preserve-access-time
--preserve-creation-time


Here is a list of sample file transfers, and the resulting metadata tags in S3.  In my example, I am uploading a file 'Capture.PNG' with the following time file time stamps.  NOTE:  My system is running on PST (GMT -8).

 

File_time_stamps.PNG

 

 

Example 1:  Transfer a file 'Capture.PNG' to object storage, using the '-p' option:

ascp -P 33001 -p Capture.PNG xfer@54.184.43.31:

Results:  The following new metadata tags are created and populated

x-amz-meta-atime
x-amz-meta-mtime

 Screenshot for example 1:

 NOTE: The time stamps are preserved in GMT.

File_time_stamps_in_S3.PNG

 

Example 2:   Transfer a file 'Capture.PNG' to object storage, using the '--preserve-modification-time' option:

ascp -P 33001 --preserve-modification-time Capture.PNG xfer@54.184.43.31:

Results:  The following new metadata tags are created and populated:

x-amz-meta-mtime

 Screenshot for example 2:

 NOTE: The time stamps are preserved in GMT.

 preserve-modification-time.PNG

 

Example 3:   Transfer a file 'Capture.PNG' to object storage, using the '--preserve-modification-time and --preserve-access-time' option:

ascp -P 33001 --preserve-modification-time --preserve-access-time Capture.PNG xfer@54.184.43.31:

Results:  The following new metadata tags are created and populated:

x-amz-meta-mtime
x-amz-meta-atime

 NOTE: This is functionally equivalent to to the '-p' option.

Screenshot for example 3:

  NOTE: The time stamps are preserved in GMT.

 File_time_stamps_in_S3.PNG

 

Example 4:   Transfer a file 'Capture.PNG' to object storage, using the '--preserve-modification-time, --preserve-access-time and --preserve-creation-time' option:

ascp -P 33001 --preserve-modification-time --preserve-access-time --preserve-creation-time Capture.PNG xfer@54.184.43.31:

Results:  The following new metadata tags are created and populated:

x-amz-meta-mtime
x-amz-meta-atime
x-amz-meta-ctime

 NOTE: This is functionally equivalent to to the '-p --preserve-creation-time' option.

Screenshot for example 4:

 

 all_3_file_stamps_preserved.PNG

 

 

  

 Results

To preserve the file time stamps when downloading, simple use the same options, but reverse the file direction.  

Example 1: In this example, I am downloading the same file to my 'scratch' folder using the "-p" option.

ascp -P 33001 -p xfer@54.184.43.31:Capture.PNG scratch/

The resulting time stamps can be seen in this screenshot:

Downloaded_file_time_stamp.PNG

 

To also preserve all three time stamps, simply include --preserve-modification-time, preserve-access-time, preserve-creation-time options.  For example:

ascp -P 33001 --preserve-modification-time --preserve-access-time --preserve-creation-time xfer@54.184.43.31:Capture.PNG scratch/

 

 

 

 

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.
Powered by Zendesk