How to configure and use Node API for parallel transfers

Overview

The 3.6 release of Aspera Enterprise Server includes a REST API for initiating parallel transfers. This new feature is an extension of the asperanode /ops/transfers API. Parallel transfers can move content between single servers or clusters, and to/from block or object storage. 

 

             Server_to_Cluster_Parallel_transfers.png                           Cluster_to_Cluster_parallel_transfers.png

 

Details

In this KB article, we describe how to interact with this new API. We are working on building this transfer initiation capability into various applications, but for the current version, it is only exposed as an API. The best way to interact with this API, is via the curl command line utility. This article walks you through setting up a server system and interacting with the API. In the cluster to cluster scenario, the Node API is designed to distribute the sessions over the cluster on the initiating side.  

Configuration

Datacenter (on premises) system

1. Install Aspera Server (3.6) on Linux 64 bit

 rpm -ivh aspera-entsrv-3.6.version-linux-64-release.rpm

2. Create a user & API access

  • Create a system user / password
# useradd xfer
# passwd xfer
(NOTE: You will be prompted to provide a password)
  •  Create node user (substitute PASSWORD for your password)
 # /opt/aspera/bin/asnodeadmin -a -u xfer -x xfer -p PASSWORD
  • Verify that the xfer account is set up
#/opt/aspera/bin/asnodeadmin -l
List of node users(s):
        user     system/transfer user                  acls
==============    =====================      ===============
        xfer                    xfer      [ ]

3. Configure the user with an Aspera document root (for example, /data)

asconfigurator -x "set_user_data;user_name,xfer;absolute,/data"

4. Identify or create a local directory for downloads and set permissions so xfer1 user can write to it. For example:

# mkdir /data/
# chmod 777 /data/

5. Additional required server configurations. Please substitute appropriately.

# asconfigurator -x "set_server_data;activity_logging,true"
# asconfigurator -x "set_server_data;server_name,YOUR_SERVER_IPADDRESS"
# asconfigurator -x "set_node_data;token_encryption_key,ENCRYPTION_KEY"

6.  Additional optional server configurations.  Please make appropriate substitutions.

# asconfigurator -x "set_server_data;transfers_multi_session_default,NUMBER"
# asconfigurator -x "set_server_data;transfers_retry_duration,TIME(Minutes)"
NOTE: You can also specify transfers_multi-session in the JSON transfer spec. The default retry duration is 20 minutes, but if you want a longer retry period, you can specify it by setting transfers_retry_duration.

Cluster system 

NOTE: The easiest way to setup a cluster in the cloud, is by using the Aspera Cluster Manager. Once you have launched your cluster, you need to create an Access Key and give that access key permissions to an S3 bucket. The server will need to have a few options enabled, as can be seen in the tuning options shown below.

Here are some server tuning options for your server's aspera.conf:

<server> 
   ...     
   <activity_logging>true</activity_logging> # Required for noded when used in cluster.
# default is 'false'
   <ssl_protocol>sslv23</ssl_protocol> # Default is tlsv1, no need call explicitly
   <multi_session_threshold_default>1</multi_session_threshold_default> # Useful in Shares to Shares transfer
   <workers>30</workers> # Number of concurrent http/s request handling
# threads. The default is 20
   <log_api_activity>true</log_api_activity> # Not related to parallel transfers
# but will enable additional logging
   <activity_retention>8h</activity_retention> # Default is 36 hours
   <activity_cleanup_interval>2h</activity_cleanup_interval> # Default is 1 hours
<transfers_retry_duration></transfers_retry_duration> # Default is 20 min. How long will transfers be
# retried? Extended each time there is activity
<transfers_retry_all_failures></transfers_retry_all_failures> # Default is false. Setting to 'true' will over
# ride FM. Transfers previously considered
# unretriable will now be retried.
   ...
</server>
<default>
     ...
     <transfer>
         <protocol_options>
             ...
             <chunk_size>67108864</chunk_size> # Needed by trapd with parallel transfers
         </protocol_options>
<multi_session_threshold_default> # File size threshold for splitting between ascp
</multi_session_threshold_default> processes. Default is -1 (see below)
     </transfer>
     ...
</default>

Tuning Details

1. multi_session_threshold_default

This setting configures the system to split files between multiple ascp processes if the file size is greater than or equal to the specified size in bytes.

Allowed settings are the following:

  • -1 : Default, which means the setting is unspecified
  •  0  : Disables multi-session transfers
  • >0 : Enables multi-session transfers for files greater than or equal to this size

For example, to split files that are greater than or equal to 100MB, you would configure this setting as follows:

<multi_session_threshold_default>104857600</multi_session_threshold_default>

To disable splitting of files altogether, you would configure the setting as follows:

<multi_session_threshold_default>0</multi_session_threshold_default>

Transfer initiation

The transfers API can be exercised via the Linux command line using the curl command. The API can be used to send and receive single files, file lists, or directories. Here is an example of the command syntax: 

# curl -k -v -X POST -d @/PATH_TO/spec.json https://API_USER:API_PASSWORD@IPADDRESS:9092/ops/transfers

Please take note of the following two substitutions:

  • for PATH_TO - provide the path to your spec file.
  • for spec.json - use the name of the spec file that you create (see examples below in Transfer specification examples)
  • for API_USER:API_PASSWORD - use the Node API user account credentials that were created in step #2 above
  • For IP_ADDRESS - use the IP address of the system initiating transfers

Transfer specification examples

This spec file is essentially the transfer recipe. Examples for sending, receiving, directories, file lists, and individual files are provided. Use your favorite text editor to create these files, for example:

 # vi /root/spec.json

Example 1. Receiving (downloading) a directory 

NOTE: The token comes from the ‘access key’ created on the cluster.

Example content for receiving an entire folder:

{
"transfer": {
"remote_host": "example.server.com",
   "remote_user": "xfer",
   "token": "Basic YRNxZXHGOmPsdWI=",
   "target_rate_kbps": 100000,
   "multi_session": 5,
   "destination_root":"/",
   "paths": [
     {
       "source": "/SOURCE_DIRECTORY/"
     }
   ],
   "ssh_port": 33001,
   "fasp_port": 33001,
   "direction": "receive",  
   "cookie": "Multipart download",
   "rate_policy_allowed": "fair",
   "rate_policy": "fair",
   "tags": {
   }
 }
}

 

Example 2. Receiving (downloading) a list of files.

To receive a list of files, use this syntax:

        "paths": [
            { "source": "/FOLDER/FILE1" },
            { "source": "/FOLDER/FILE1” },
            { "source": "/FILE3" }
        ],

Example 3.  Receiving (downloading) a single file.

To receive a specific file, use this syntax:

        "paths": [
            { "source": "/FILE3" }
        ],

Example 4.  Sending (uploading) a directory 

Example content for sending a directory:

{
  "transfer": {
    "remote_host": "server.example.com",
    "remote_user": "xfer",
    "token": "Basic TERQNTBhZzZ0Y0dIbGJqcUJzSU06NlJRTGx4VERURF9ndHF5bVd1OGlvUTBLeU5mR3kwbFBiNXBNWnA4b0dreEY=",
    "target_rate_kbps": 80000,
    "multi_session": 12,
    "destination_root":"/",
    "paths": [
      {
        "source": "/SOURCE_DIRECTORY/"
      }
    ],
    "ssh_port": 33001,
    "fasp_port": 33001,
    "direction": "send",
    "cookie": "Transfer with 12 multipart",
    "rate_policy_allowed": "fair",
    "rate_policy": "fair",
    "tags": {
    }
  }
}

Example 5.   A list of all spec file options

{
  "transfer": {
    "remote_host": "server.example.com",
    "remote_user": "xfer",
    "token": "Basic TERQNTBhZzRUR5mR3kwbFBiNXBNWnA4b0dreEY=",
    "target_rate_kbps": 80000,
    "multi_session": 12,
    "destination_root":"/",
    "paths": [
      {
        "source": "/SOURCE_DIRECTORY/"
      }
    ],
    "ssh_port": 33001,
    "fasp_port": 33001,
    "http_fallback": false,
    "http_fallback_port": 80,
    "https_fallback_port": 443,
    "direction": "send",
    "overwrite": "always",
    "source_root": null,
    "cipher_allowed": "none",
    "content_protection": null,
    "cookie": "Transfer 4x multipart",
    "rate_policy_allowed": "fair",
    "rate_policy": "fair",
    "tags": {
    }
  }
}

Advanced API options

You should be able to specify the following attribute in json as a sibling to other attributes:

"overwrite" : "a_str_literal",

where a_str_literal can be one of the following possible values:

  • never
  • always
  • diff
  • older
  • diff+older

Troubleshooting tips

1. Are there any ascp processes running?

# ps exff | grep ascp 

 2. Is the asperanoded service running, and accessible on port 9092?

Check that the service is running:
# service asperanoded status
Restart the asperanoded service:
# service asperanoded restart
Verify that it is listenting on port 9092:
# netstat -anp | grep 9092
Connect to the node using the API account:
# curl -ik https://xfer1:PASSWORD@IP_ADDRESS:9092/info

3. What do the logs files say when you execute the command?

# tail -f /var/log/messages

4. How to query node for live transfers

5. How to query node for Access Key setup (e.g - list, info)

6. What is the cause of the following error message?

# "error" : { "code" : 400, "reason" : "Bad Request", "user_message" : "Storage root is not configured." }

Answer: Most likely you have updated the aspera.conf file, but not restarted the asperanoded service. Please restart the asperanoded service and try again.

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.
Powered by Zendesk