Persistent storage options for Aspera On Demand in Softlayer

Summary


This article is written to help you configure persistent storage options for Aspera on Demand for Softlayer.  Setting the Document Root to a SWIFT container is different than setting it to a filesystem.  Before we can configure the Document root, we first need to determine the URI that represents the Object Storage.

 

Details

Determine Document Root for URI for SWIFT Object Storage

This section describes how to configure Aspera Server (Server On Demand, Application Platform On Demand) to use  SWIFT object storage as the primary storage location for the transfer (e.g. system) users.  Each system user that uses SWIFT will need to have their 'Docroot' configured to point to object storage using a custom Object Storage URI.  The URI will contain a few elements of data, including the Object Storage authentication endpoint, the Object Storage user and API Key, as well as any containers.  

  1. Obtain the Object Storage Authentication endpoint information.   When the object storage is created the credentials are shown as follow - for the storage cluster located in Dallas-5.   This information can be obtained by looking at the details of your object storage from within the Softlayer portal.  NOTE: The Softlayer portal will provide both Public and Private URLs.  We strongly urge you to use the private URLs, to avoid excessive egress costs and load on your network interfaces.

Object Storage information Authentication Endpoint
Private: https://dal05.objectstorage.service.networklayer.com/auth/v1.0/

  1. User and Access Key credentials:  Get your Softlayer username and the API Key (e.g. password).  This information can also be obtained from within the Softlayer portal.

Username: IBMOS303456-2:someuser 
API Key (Password): af879bf5737b3e7f98919d98f524c52d62771a736066b39606968fef4db78f77

  1. Configure the Document Root for your user.   The syntax for the URI is as follows:

swift://ACCOUNT:USERNAME:API_KEY@PRIVATE_SOFTLAYER_AUTHENTICATION_HOST/CONTAINER

  1. Here is an example of what the Document root setting in aspera.conf should look like: 

 

...  
 <file_system>
     <access>
         <paths>
              <path>

             <absolute>swift://IBMOS303456-2%3Asomeuser:af879bf5737b3e7f98919d98f524c52d62771a736066b39606968fef4db78f77@dal05.objectstorage.service.networklayer.com/test</absolute>

          </path>
       </paths>
    </access>
</file_system>

...

Note: 

  1. The URI does not include the /auth/v1.0/. This has to be defined later.
  2. Both above URIs (public and private)  would work, but the best is to use the private one that uses the internal network (on eth0) to communicate with the storage.
  3. Note that the ":" in the user name must be url encoded ("%3A").

 

 

Configuring the Document Root - Console

There are a few ways to configure the Document Root, Graphical Interface: Use the Aspera Console; Command Line: use the 'asconfigurator' command; VI: Edit aspera.conf directly.

   In the Aspera Console, navigate to the Nodes -> Accounts -> Edit where you can edit the docroot of an existing account.  You can also choose to create a new account (there is a 'new' button on the previous screen).   On the Edit screen, you need to click on 'overwrite' to be able to paste the SWIFT docroot.  See screenshot.

 

Creating_a_docroot_in_Console.png

 

 

Configuring the Document Root - Command line

If you do not have a web UI, or just prefer to use the command line,  you can always use the utility 'asconfigurator'.  Below is an example of how to add a document root to an existing user 'xfer'.   A full KB article on how to use asconfigurator for setting up users can be found HERE.

 asconfigurator -x "set_user_data;user_name,xfer;absolute,swift://IBMOS303456-2%3Asomeuser:af879bf5737b3e7f98919d98f524c52d62771a736066b39606968fef4db78f77@dal05.objectstorage.service.networklayer.com/test"

 

 

 

Additional settings - Auth Type

For systems that were manually installed and used on Softlayer Openstack, the following settings must also be changed.

  1. Edit the Aspera Trapd main configuration file, using your favorite text editor

/opt/aspera/etc/trapd/swift.properties

  1. Uncomment and change the auth property configuration, as follows:
# Auth path (looks as a virtual path)
# Default is '/v1.0'
aspera.swift.endpoint.auth-path=/auth/v1.0
  1. Restart the Aspera Trapd service (when logged in as root)

# service asperatrapd restart

 

 

Additional settings - Static Large Object Support

If you are using Aspera on Demand to connect to a SWIFT object storage which is < 1.12 release version, then you need to make the following change

  1. Update the trapd init script.

#/etc/init.d/asperatrapd

  1. Make the following adjustments in method trap_init_start() by adding the colored line below:
# disable checks done on open-write for small files
#PROPS="$PROPS -Daspera.session.upload.start-check-small-file.enabled=false"
PROPS="$PROPS -Daspera.swift.session.upload.large-object.dynamic=true" 
  1. Restart the Aspera Trapd service

# service asperatrapd restart

 

 

 

 

Have more questions? Submit a request

1 Comments

  • Avatar
    Jonathan Solomon

    Please note:

    In Additional Settings - Auth Type, you must add "auth" to the un-commentated line.

Please sign in to leave a comment.
Powered by Zendesk