Getting started with Aspera On Demand for Google Cloud

Avatar
Crystal Yeung

Overview

This article provides step-by-step instructions on how to set up Aspera on Google Cloud and enable data transfer to and from Google Cloud Storage. 

 

Prerequisites

This article assumes the following:

  • You have an account with Google Cloud
  • You have created a bucket on Google Cloud Storage
  • You know how to launch VM Instances on Google Cloud.  See this article for instructions.
  • You know how to generate a ssh key pair
  • You are able to ssh into your Google instance as root.  See this article for instructions.

 

System Requirements

  • Linux Kernel must be at 2.6.34 or higher.  Use "uname -a" to verify

 

Procedures

1) Login to Google Developers Console 

https://cloud.google.com/console/project

 

2) Select the relevant project 

If you don't have a project, create a new project.

create_project.jpg

 

3) Generate a ssh key pair

Run ssh-keygen to create a ssh key pair on a computer from which you would use to connect to an instance that you are about to generate.

# ssh-keygen -t rsa

 

4) Create  a new instance

Go to Dashboard --> Take a VM quickstart.   For more detail, please see this article.

create_vm1.jpg

           

Under "VM instances", enter Name, Zone, Machine type, and Boot disk. 

 

 

Under "Identity and API access", make sure "Enable Compute Engine service account" and "Set access for each API" are both selected. Ensure that you have the following if using Google Cloud storage:

a) User info = Enabled

b) Compute Engine = Read Write

c) Storage = Full

Add the SSH key you created in Step 3) in the  "SSH Keys" section. For Firewall requirements, please refer to Aspera product guide and read this article. With your customer login credentials, Aspera product guide can be downloaded from our website. http://downloads.asperasoft.com/downloads. Once you fill out all the required fields, click the "Create" button to create your instance.

 

5) SSH into your instance as root and configure the server

a) Using the user and private key you created in Step 3) and 4) to connect to your instance via ssh. 

Once a connection is established, elevate the privileges to root.

  # ssh username@gc.instance.ip.address
  # sudo -i

        

b) Update sshd_config to enable port 33001, password authentication and root login

It is well known that SSH servers listen for incoming connections on TCP Port 22. As such, Port 22 is subject to countless, unauthorized login attempts by hackers who are attempting to access unsecured servers. A highly effective deterrent is to simply turn off Port 22 and run the service on a seemingly random port above 1024 (and up to 65535). To standardize the port for use in Aspera transfers, we recommend using TCP/33001.

# vi /etc/ssh/sshd_config

To enable TCP/33001, add Port 33001 in your/etc/ssh/sshd_configfile (where SSHD is listening on both ports). As demonstrated below.  SSHD is capable of listening on multiple ports.

...
Port 22
Port 33001

In addition, you need to enable password authentication. 

...
# Authentication
...
PasswordAuthentication yes

Restart sshd after making the changes.     

   # service sshd restart

c) Check local Firewall

Make sure your local Firewall supports Aspera or is disabled.  For Firewall requirements, please refer to your product guide and read this article. 

d) Check /etc/hosts file

Verify /etc/hosts file contains an entry for 127.0.0.1 localhost.

e) Disable SELINUX. 

NOTE: If done incorrectly, your system may become not bootable.

f) Create system user accounts and set passwords

Your Aspera server uses your system accounts to authenticate connections. These system accounts must be added before attempting an Aspera transfer.

g) Make sure the Google firewall is not blocking the ports required for FASP.  Click here for article.

 

 

6) Install Aspera v.3.4.x or higher

Use your customer login credentials to download Aspera installer and product guide from our website. http://downloads.asperasoft.com/downloads.  Follow the instructions documented in the product guide to install and set up Aspera.

 

7) Enable Entitlement

Once Aspera Enterprise Server is successfully installed, run the ALEE command to register the information. In example below, foo is the Customer ID and d1234567-7fac-409d-9200-e2cb1234567c the Entitlement ID

# /opt/aspera/bin/asalee-config.sh enable
# service asperanoded restart
# /opt/aspera/bin/alee-admin register foo d1234567-7fac-409d-9200-e2cb1234567c

 

NOTE: Faspex v.3.7.8+, Shares v.1.7.3+ and Console v.2.3.2+ have its own entitlement steps.  If you installed any of these products, please see the following instructions.

To entitle Faspex v.3.7.8 or higher

# export RAILS_ENV=production 
# asctl faspex:rake entitlement:config_license_server EL_KEY="d1234567-7fac-409d-9200-e2cb1234567c" EL_CUSTOMER_ID="foo"

To entitle Shares v.1.7.3 or higher

# /opt/aspera/shares/bin/run bash -c 'cd /opt/aspera/shares/u/shares && RAILS_ENV=production bundle exec rake aspera:ami:entitlement:config_license_server EL_KEY="d1234567-7fac-409d-9200-e2cb1234567c" EL_CUSTOMER_ID="foo"'

To entitle Console v.2.3.2 or higher

# cd /opt/aspera/console/
# export RAILS_ENV=production
# export PATH=/opt/aspera/common/ruby/bin:$PATH
# rake aspera:ami:entitlement:config_license_server EL_KEY="d1234567-7fac-409d-9200-e2cb1234567c" EL_CUSTOMER_ID="foo"

 

8) Enable trapd and set the MTU size to 1460

Enable trapd if you plan to transfer data from and to Google Cloud Storage.

# /opt/aspera/bin/astrap-config.sh  enable
    # asconfigurator -x "set_node_data;transfer_protocol_options_datagram_size,1460"

9) Enable transfer user to Google Cloud Storage

         Use asconfigurator to set docroot for the transfer user.  In this example, the transfer user is "xfer

# asconfigurator -F "set_user_data;user_name,xfer;absolute,gs://storage.googleapis.com/your_bucket/your_path/"

then restart noded

# service asperanoded restart

        NOTE: If you need to transfer to a storage that belongs to a different project under the same account, please see this article for details.

10) Verify docroot for the transfer user

Run asuserdata to verify user docroot.  Assume "xfer" is the transfer user.

# /opt/aspera/bin/asuserdata -u xfer

output for "docroot option set" should be the following

  ...

  docroot option set:
      canonical_absolute=gs://storage.googleapis.com/your_bucket/your_path/
      canonical_show_as=/
      absolute: "gs://storage.googleapis.com/your_bucket/your_path/"

  ...

 

 11) Run test transfers

Use Aspera Client, P2P or Enterprise Server (or Connect plug-in if you install Shares/Faspex/Connect Server on your instance) to run test transfers with your Aspera server on Google Cloud.

(On your client machine) Initiate a transfer to Aspera server on Google Cloud

To do so, run the following command on your client machine (wherexferis our example transfer user):

$ ascp -P 33001 -T --policy=fair -l 10000 /client-dir/files xfer@gc.instance.ip.address:/

Please refer to the product guide if you need more details on how to set up test transfers.

manual.jpg

If your set up is successful, you should see files in your Google Cloud Storage bucket after pushing data to the server. 

 

 

 

Powered by Zendesk