Overview
This article provides step-by-step instructions on how to set up Aspera on Google Cloud and enable data transfer to and from Google Cloud Storage.
Prerequisites
This article assumes the following:
- You have an account with Google Cloud
- You have created a bucket on Google Cloud Storage
- You know how to launch VM Instances on Google Cloud. See this article for instructions.
- You know how to generate a ssh key pair
- You are able to ssh into your Google instance as root. See this article for instructions.
System Requirements
- Linux Kernel must be at 2.6.34 or higher. Use "uname -a" to verify
Procedures
1) Login to Google Developers Console
https://cloud.google.com/console/project
2) Select the relevant project
If you don't have a project, create a new project.
3) Generate a ssh key pair
Run ssh-keygen to create a ssh key pair on a computer from which you would use to connect to an instance that you are about to generate.
# ssh-keygen -t rsa
4) Create a new instance
Go to Dashboard --> Take a VM quickstart. For more detail, please see this article.
Under "VM instances", enter Name, Zone, Machine type, and Boot disk.
Under "Identity and API access", make sure "Enable Compute Engine service account" and "Set access for each API" are both selected. Ensure that you have the following if using Google Cloud storage:
a) User info = Enabled
b) Compute Engine = Read Write
c) Storage = Full
Add the SSH key you created in Step 3) in the "SSH Keys" section. For Firewall requirements, please refer to Aspera product guide and read this article. With your customer login credentials, Aspera product guide can be downloaded from our website. http://downloads.asperasoft.com/downloads. Once you fill out all the required fields, click the "Create" button to create your instance.
5) SSH into your instance as root and configure the server
a) Using the user and private key you created in Step 3) and 4) to connect to your instance via ssh.
Once a connection is established, elevate the privileges to root.
# ssh username@gc.instance.ip.address
# sudo -i
b) Update sshd_config to enable port 33001, password authentication and root login
It is well known that SSH servers listen for incoming connections on TCP Port 22. As such, Port 22 is subject to countless, unauthorized login attempts by hackers who are attempting to access unsecured servers. A highly effective deterrent is to simply turn off Port 22 and run the service on a seemingly random port above 1024 (and up to 65535). To standardize the port for use in Aspera transfers, we recommend using TCP/33001.
# vi /etc/ssh/sshd_config
To enable TCP/33001, add Port 33001 in your/etc/ssh/sshd_configfile (where SSHD is listening on both ports). As demonstrated below. SSHD is capable of listening on multiple ports.
... Port 22 Port 33001
In addition, you need to enable password authentication.
...
# Authentication
...
PasswordAuthentication yes
Restart sshd after making the changes.
# service sshd restart
c) Check local Firewall
Make sure your local Firewall supports Aspera or is disabled. For Firewall requirements, please refer to your product guide and read this article.
d) Check /etc/hosts file
Verify /etc/hosts file contains an entry for 127.0.0.1 localhost.
e) Disable SELINUX.
NOTE: If done incorrectly, your system may become not bootable.
f) Create system user accounts and set passwords
Your Aspera server uses your system accounts to authenticate connections. These system accounts must be added before attempting an Aspera transfer.
g) Make sure the Google firewall is not blocking the ports required for FASP. Click here for article.
6) Install Aspera v.3.4.x or higher
Use your customer login credentials to download Aspera installer and product guide from our website. http://downloads.asperasoft.com/downloads. Follow the instructions documented in the product guide to install and set up Aspera.
7) Enable Entitlement
Once Aspera Enterprise Server is successfully installed, run the ALEE command to register the information. In example below, foo is the Customer ID and d1234567-7fac-409d-9200-e2cb1234567c the Entitlement ID
# /opt/aspera/bin/asalee-config.sh enable
# service asperanoded restart
# /opt/aspera/bin/alee-admin register foo d1234567-7fac-409d-9200-e2cb1234567c
NOTE: Faspex v.3.7.8+, Shares v.1.7.3+ and Console v.2.3.2+ have its own entitlement steps. If you installed any of these products, please see the following instructions.
To entitle Faspex v.3.7.8 or higher
# export RAILS_ENV=production
# asctl faspex:rake entitlement:config_license_server EL_KEY="d1234567-7fac-409d-9200-e2cb1234567c" EL_CUSTOMER_ID="foo"
To entitle Shares v.1.7.3 or higher
# /opt/aspera/shares/bin/run bash -c 'cd /opt/aspera/shares/u/shares && RAILS_ENV=production bundle exec rake aspera:ami:entitlement:config_license_server EL_KEY="d1234567-7fac-409d-9200-e2cb1234567c" EL_CUSTOMER_ID="foo"'
To entitle Console v.2.3.2 or higher
# cd /opt/aspera/console/
# export RAILS_ENV=production
# export PATH=/opt/aspera/common/ruby/bin:$PATH
# rake aspera:ami:entitlement:config_license_server EL_KEY="d1234567-7fac-409d-9200-e2cb1234567c" EL_CUSTOMER_ID="foo"
8) Enable trapd and set the MTU size to 1460
Enable trapd if you plan to transfer data from and to Google Cloud Storage.
# /opt/aspera/bin/astrap-config.sh enable
# asconfigurator -x "set_node_data;transfer_protocol_options_datagram_size,1460"
9) Enable transfer user to Google Cloud Storage
Use asconfigurator to set docroot for the transfer user. In this example, the transfer user is "xfer
# asconfigurator -F "set_user_data;user_name,xfer;absolute,gs://storage.googleapis.com/your_bucket/your_path/"
then restart noded
# service asperanoded restart
NOTE: If you need to transfer to a storage that belongs to a different project under the same account, please see this article for details.
10) Verify docroot for the transfer user
Run asuserdata to verify user docroot. Assume "xfer" is the transfer user.
# /opt/aspera/bin/asuserdata -u xfer
output for "docroot option set" should be the following
...
docroot option set:
canonical_absolute=gs://storage.googleapis.com/your_bucket/your_path/
canonical_show_as=/
absolute: "gs://storage.googleapis.com/your_bucket/your_path/"...
11) Run test transfers
Use Aspera Client, P2P or Enterprise Server (or Connect plug-in if you install Shares/Faspex/Connect Server on your instance) to run test transfers with your Aspera server on Google Cloud.
(On your client machine) Initiate a transfer to Aspera server on Google Cloud
To do so, run the following command on your client machine (wherexferis our example transfer user):
$ ascp -P 33001 -T --policy=fair -l 10000 /client-dir/files xfer@gc.instance.ip.address:/
Please refer to the product guide if you need more details on how to set up test transfers.
If your set up is successful, you should see files in your Google Cloud Storage bucket after pushing data to the server.
0 Comments