Aspera on Demand with Cargo

Cargo on APOD

This article provides the steps on how to install and configure Cargo on Aspera on Demand for AWS so that Cargo will download packages and deposit them into an S3 bucket.

Pre-requisites

  1. SOD/APOD up and running
  2. You have a working account on a Faspex Server
    Note: In the demo below, I use the ‘cargo’ account on Faspex.asperasoft.com 
  3. A transfer account runs in bash shell with S3 document root.  Incoming Transfers authorization should set to "allow" for this transfer account.

Steps

1) Log into your Aspera server as the user root.  If you don't know how, please read this article

2) Download and install Cargo 64 bit:

wget http://download.asperasoft.com/download/sw/cargod/1.3/aspera-cargod-1.3.0.80012-linux-64.rpm 

3) Install RPM

rpm –ivh aspera-cargod-1.3.0.80012-linux-64.rpm

4) Configure Cargo

vi /opt/aspera/cargod/etc/asperacargo.conf

Here is a sample minimal required Configuration file:

<?xml version="1.0" encoding="UTF-8"?>
<conf version="1">
    <accounts>
        <account>
            <user>cargo</user>
            <password>cargo_password</password>
            <host>https://faspex.asperasoft.com/aspera/faspex</host>
            <download_folder>/</download_folder>
            <passphrase></passphrase>
        </account>
    </accounts>
    <general>
        <check_interval>5</check_interval>
    </general>
</conf>
NOTE: The password will be obfuscated once Cargo is invoked.

5) Verify Cargo works as expected

a) Launch Cargo

/opt/aspera/cargod/bin/asperacargod &

b) Send a Faspex package to user cargo via Faspex Web UI
c) Check whether or not the package is automatically downloaded by Cargo

NOTE: You should see a new folder with  the format "user'@'Server' in the / of your file system.

6) Once you confirm Cargo is set up correctly, change the following permissions on your file system so that Cargo can be run as non root user:

chmod 777 /opt/aspera/cargod/var
chmod 777 /opt/aspera/cargod/var/filelists
chmod 777 /opt/aspera/cargod/var/log
chmod 666 /opt/aspera/cargod/var/log/*

7) Create a new transfer user (e.g. asp1) from your APOD Console  and set asp1 runs in bash shell.

NOTE:  To set the users shell, you will need to edit /etc/passwd

8) Specify an S3 docroot for your user (e.g. asp1) in the Cargod aspera.conf as well as the main aspera.conf file

# vi /opt/aspera/cargod/etc/aspera.conf
NOTE: Once this file has been edited, you need to add a similar entry into the main aspera.conf too.
# vi /opt/aspera/etc/aspera.conf

Here is a sample of a working aspera.conf with some S3 ID and bucket substitutions:

<?xml version='1.0' encoding='UTF-8'?>
<CONF version="2">
<default>
    <file_system>
        <storage_rc>
            <adaptive>
                true
            </adaptive>
        </storage_rc>
    </file_system>
</default>
<aaa>
    <realms>
        <realm>
            <users>
                <user>
                    <name>asp1</name>
                    <file_system>
                        <access>
                            <paths>
                                <path>
                                    <absolute>s3://S3SECRETKEY:S3ACCESSID@s3.amazonaws.com/BUCKETNAME/</absolute>
                                    <read_allowed>true</read_allowed>
                                    <write_allowed>true</write_allowed>
                                    <dir_allowed>true</dir_allowed>
                                </path>
                            </paths>
                        </access>
                    </file_system>
                </user>
            </users>
        </realm>
    </realms>
</aaa>
</CONF>

9) Set the permissions on asperacargo.conf so that non root users have access:

chmod 666 /opt/aspera/cargod/etc/asperacargo.conf

10) Link ascp to the system default ascp

mv /opt/aspera/cargod/bin/ascp /opt/aspera/cargod/bin/ascp.ORIG
ln –s /opt/aspera/bin/ascp /opt/aspera/cargod/bin/ascp

11) Login as the user with the S3 document root (e.g. asp1)  In this example, you can see that I switch from root to the user 'asp1'.  

NOTE: If the sudo command below fails, please verify that you have set the asp1 user to use the bash shell (rather than aspshell - which prevents users from accessing the command line on the system).
# sudo su – asp1

12) In order to have ascp respects the local document root configurations, set the environment variable: ASPERA_SCP_DOCROOT by adding the following to the user's ~/.bash_profile.  Log out as asp1 and log back in as asp1 to activate the environment variable.

ASPERA_SCP_DOCROOT=1
export ASPERA_SCP_DOCROOT

13) Create a default landing folder in your S3 docroot, with the syntax user@host. In my example, I have ‘cargo@faspex.asperasoft.com’. You can use S3 Management Console in AWS,  Aspera SCP GUI (P2P or ES), or the S3 browser to create this folder.

14) Remove the database file and start Cargo in the background.  NOTE: Cargo must be run by the transfer user with S3 docroot.

# rm /opt/aspera/cargod/var/asperacargod.dat*
# /opt/aspera/cargod/bin/asperacargod &
Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.
Powered by Zendesk