Adding Additional S3 Buckets to OnDemand Shares

Shares OnDemand Concepts:

What are Nodes, API Users, S3 Buckets, and S3 Docroots (and how are they related)?

An API user is a system user with a docroot that is used for API level access, like listing of files and initiating/authenticating transfers.

A Node in Shares is a System with Aspera Connect Server (Enterprise Server that is Web Plugin enabled) and a docroot on the system as represented by an API User. More accurately, a Shares Node is a system that Shares shares-out as represented by an API User.

S3 Buckets are discrete storage containers on the S3 object-store. Buckets are not like a file-system where there is a root-level or namespace. Each is a separate container. As such, there is no way to create a single Aspera Docroot that encapsulates multiple buckets.

S3 Docroots map the root-level of a transfer destination/source to a single S3 bucket.

Since the API User views storage as a docroot, and an S3 docroot only maps to a single bucket, having Shares view multiple S3 buckets requires unique/individual API users (and nodes in Shares) for each bucket.

Setting up additional S3 docroots and Nodes

To simplfy the Shares OnDemand initial deployment, 2 API users are created. The xfer2 API user is for the initial S3 bucket. To use more than the one S3 bucket, additional S3 buckets need to be associated with new API users. The following steps show how to create API users with S3 docroots, then create the appropriate node. For these steps, the API user and local user used will both be xfer3. The API user and local user do not need to be the same, but for simplicity, they will be.

The S3 location is given in the form of a [URL] (http://en.wikipedia.org/wiki/Uniform_resource_locator). In general, the format is:

s3://AWSAPIUSER:AWSAPIKEY@s3.amazonaws.com/BUCKETNAME

Where

  • AWSAPIUSER is the AccessKeyID found on the AWS account/IAM user (https://forums.aws.amazon.com/thread.jspa?threadID=49738);
  • AWSAPIKEY is the SecretAccessKey found on the AWS account/IAM user (https://forums.aws.amazon.com/thread.jspa?threadID=49738);
  • BUCKETNAME is the S3 bucket for the data.

Please note that the AWSAPIKEY/SecretAccessKey can have special characters that do not conform to the URL syntax. These need to be escaped using [percent encoding] (http://en.wikipedia.org/wiki/Percent-encoding).

  1. The first step is to create a local user on the OS that will have the S3 docroot. The local user will be xfer3 as noted above.

    # useradd xfer3
    # mkdir /home/xfer3/.ssh
    # cp /opt/aspera/var/aspera_id_dsa.pub /home/xfer3/.ssh/authorized_keys
    # chown -R xfer3:xfer3  /home/xfer3/.ssh
    # chmod 700 /home/xfer3/.ssh
    
  2. The Aspera configuration (/opt/aspera/etc/aspera.conf) needs to be configured to have the S3 docroot for the xfer3 user just created, and the auth method should be set to token:

    # asconfigurator -F "set_user_data;user_name,xfer3;absolute,s3://AWSAPIUSER:AWSAPIKEY@s3.amazonaws.com/BUCKETNAME;authorization_transfer_in_value,token;authorization_transfer_out_value,token"
  3. Now an API user is created to map the S3 docrooted local user. Note that a password is set for the API user. The password for pre-created xfer and xfer2 are the InstanceID, but subsequent passwords can be anything.

    # /opt/aspera/bin/asnodeadmin -a -u xfer3 -p a_secret_node_api_password -x xfer3 
  4. Then the asperanoded service needs to be restarted for the docroot to take effect.

    # service asperanoded restart
  5. Log into Shares as Admin, and add a new Node by clicking on the ‘+’ symbol next to the Node menu heading. S3-Step1-LoggedIn.png

  6. Add the following fields, as depicted in the screenshot below, including the API password. (the default password is the instance ID.)S3-Step2a-NewNode.png
  7. Browse the newly added node (S3 Storage) by clicking on it, and confirm that you can now see the contents of your S3 bucketS3-Step3a-Result.png
Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.
Powered by Zendesk