Aspera Sync to S3

Summary

This article summarizes some options for using the async application when either the source or destination is S3 object storage.  At the writing of this article, async for S3 is not released in production yet.  This article is a document for customers who are interested to test this capability.  In order to use this feature, you will need to get an updated async and ascp binaries that have support for this feature.  

Details

Here are the modes of operation that work

Non-continuous mode:  

             PUSH, PULL and BIDI mode work fully

Continuous mode: 

             PUSH: works fully (from local disk to S3)

             PULL: Only changes during first scan work. (No notifcations in S3)

             BIDI: This is a combo of PUSH and PULL

Preparation

Install a current generation of Aspera Enterprise Server for Linux 64 on a local system.  Get access to your Aspera On Demand (any recent release will do).  Obtain a new async and ascp binary from your Aspera technical contact.  Copy the async binary to both source and destination server.  Copy the ascp binary to the server in AWS.

Configuration

In order to use async with S3, you will need to configure your Server On Demand as follows:

  1. Create a system account with an S3 docroot. Specify the bucket only, not the sub folders.  Assuming you are using IAM roles, you do not need to provide any S3 credentials
    # asconfigurator -x "set_user_data;user_name,USERNAME;absolute,s3://s3.amazonaws.com/BUCKETNAME;"
  2. Configure async to use a specific db directory
    # asconfigurator -x "set_node_data;async_db_dir,/data/db"
    # asconfigurator -x "set_node_data;async_log_dir,/data/logs"

Run

Here is a sample script that does a PUSH, using SSH keys from /SOURCE/FOLDER:

#!/bin/bash
#

async \
-L /LOG/FOLDER \
-N JOB_NAME \
-d /SOURCE/FOLDER \
-r USERNAME@HOST:/SUBFOLDER \
-i /root/.ssh/SSH_PRIVATE_KEY \
-K push \
-a fair \
-l 100m \
-c none -k none --pending-max=10 --cooloff=0 \
-b /data/db ${1}
Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.
Powered by Zendesk