How to use command line to transfer data to and from Azure

How to use command line to transfer files to and from Azure

This article provides a set of instructions on how to use the ascp command line from your Aspera machine to upload files to and download files from your storage on Azure.

 

Prerequisites:

  • You have a running Aspera on Demand on Azure
  • You have created storage on Azure.
  • You have your Azure storeage credentials
  • You have the user id and password of Azure Aspera on Demand host

Preparations:

  1. Set your storage password as an environment variable. Remember: storage password must be URI encoded on versions prior to 3.5.x 

#  export ASPERA_DEST_PASS=password_of_your_Azure_storage

           On windows systems, you would use the 'set' command:

    C:\set ASPERA_DEST_PASS=password_of_your_Azure_storage

  

     2.   Set the password of your Azure Aspera on Demand host as an environment variable. 

#  export ASPERA_SCP_PASS=password_of_your_Azure_AoD_host

        On Windows, you would use 'set' command:

   C:\set ASPERA_SCP_PASS=password_of_your_Azure_AoD_host

File Upload examples: 

    1.   CLI transfer from local system to Aspera on Demand on Azure BLOB storage:

CLI_transfer_to_Azure.png

 

# ascp  -P33001 -O33001 --user=USER_ID_FOR_AOD_HOST --host=ASPERA_ON_DEMAND_HOST --mode=send  /path/source_file azu://Azure_STORAGE_ACCOUNT_NAME@blob.core.windows.net/YOUR_BLOB_STORAGE_CONTAINER

Screen_Shot_2015-04-08_at_11.37.14_AM.png

 

   2.  CLI transfers of files from S3 using Aspera on Demand in AWS to Aspera on Demand in Azure.  

CLI_Transfer_AWS_to_Azure.png

 NOTE: In this scenario, a few additional preparation steps are required.  In your AWS instance, move the user with S3 docroot from aspshell to bash shell, set environment variables, then run the following ascp command with appropriate substitutions.  Once you finish uploading your files, move the user back to aspshell.  See the following example for details.

# ascp  -P33001 -O33001 --user=USER_ID_FOR_AOD_HOST --host=ASPERA_ON_DEMAND_HOST --mode=send  s3://s3.amazonaws.com/S3_bucket/source_file azu://USER_ID_FOR_STORAGE_ACCOUNT_NAME@blob.core.windows.net/YOUR_BLOB_STORAGE_CONTAINER

 

For example, user xfer3 is docrooted to s3://s3.amazonaws.com/nabdemo/

[root@ip-10-255-1-76 ~]# usermod -s /bin/bash xfer3
[root@ip-10-255-1-76 ~]# su - xfer3
[xfer3@ip-10-255-1-76 ~]$ export ASPERA_DEST_PASS=yyyyyyyyyyyyyyyyyyyyyyyyyy
[xfer3@ip-10-255-1-76 ~]$ export ASPERA_SCP_PASS=xxxxxxxxxxxxxxxxxxxxxx
[xfer3@ip-10-255-1-76 ~]$ ascp -P33001 -O33001 --user=xxxxxxxxxxxx --host=aa333333333.azure.xxxxxxxx.com --mode=send s3://s3.amazonaws.com/nabdemo/tinyfile0004 azu://storagename@blob.core.windows.net/abc
[xfer3@ip-10-255-1-76 ~]$ exit
[root@ip-10-255-1-76 ~]# usermod -s /bin/aspshell xfer3

File Download examples:

    1.   CLI download from Aspera on Azure BLOB storage.

CLI_transfer_from_Azure.png

 

# ascp  -P33001 -O33001 --user=USER_ID_FOR_AOD_HOST --host=ASPERA_ON_DEMAND_HOST --mode=recv  azu://USER_ID_FOR_STORAGE_ACCOUNT_NAME:password_for_azure_storage_url_encoded@blob.core.windows.net/YOUR_BLOB_STORAGE_CONTAINER/folder/filename /path/at/destination

 NOTE:  You can download multiple files or folders by using the --file-list option.  To use the --file-list command line option, you need to specify the azu: host and path information in the file list itself.  Here is an example of the command line syntax and the file list itself.

# ascp  -P33001 -O33001 --user=USER_ID_FOR_AOD_HOST --host=ASPERA_ON_DEMAND_HOST --mode=recv  --file-list=file_list /path/at/destination

a sample of file_list, file_list. Note how the full azure path for each file/folder.

azu://USER_ID_FOR_STORAGE_ACCOUNT_NAME@blob.core.windows.net/YOUR_BLOB_STORAGE_CONTAINER/folder/file1
azu://USER_ID_FOR_STORAGE_ACCOUNT_NAME@blob.core.windows.net/YOUR_BLOB_STORAGE_CONTAINER/folder/file2
azu://USER_ID_FOR_STORAGE_ACCOUNT_NAME@blob.core.windows.net/YOUR_BLOB_STORAGE_CONTAINER/folder/folder1

 

2.  CLI transfers of files from S3 using Aspera on Demand in AWS to Aspera on Demand in Azure.   

CLI_Transfer_AWS_from_Azure.png 

NOTE: In this scenario, a few additional preparation steps are required.  In your AWS instance, move the user with S3 docroot from aspshell to bash shell, set environment variables, then run the following ascp command with appropriate substitutions.  Once you finish uploading your files, move the user back to aspshell.  See the following example for details.

# ascp  -P33001 -O33001 --user=USER_ID_FOR_AOD_HOST --host=ASPERA_ON_DEMAND_HOST --mode=recv azu://USER_ID_FOR_STORAGE_ACCOUNT_NAME@blob.core.windows.net/YOUR_BLOB_STORAGE_CONTAINER/folder/filename s3://s3.amazonaws.com/S3_bucket/

 

For example, user xfer3 is docrooted to s3://s3.amazonaws.com/nabdemo/

[root@ip-10-255-1-76 ~]# usermod -s /bin/bash xfer3
[root@ip-10-255-1-76 ~]# su - xfer3
[xfer3@ip-10-255-1-76 ~]$ export ASPERA_DEST_PASS=yyyyyyyyyyyyyyyyyyyyyyyyyy
[xfer3@ip-10-255-1-76 ~]$ export ASPERA_SCP_PASS=xxxxxxxxxxxxxxxxxxxxxx
[xfer3@ip-10-255-1-76 ~]$ ascp -P33001 -O33001 --user=AAAAAAAAAAAAAAAAAAAA  --host=cc1111111111.azure.aaaaaaaaaaaaaa.com --mode=recv  azu://storagename@blob.core.windows.net/abc/test  s3://s3.amazonaws.com/nabdemo/
[xfer3@ip-10-255-1-76 ~]$ exit
[root@ip-10-255-1-76 ~]# usermod -s /bin/aspshell xfer3
Have more questions? Submit a request

1 Comments

Please sign in to leave a comment.
Powered by Zendesk