Recommended configuration for uploading small files to object store

Summary

This article is for customers who need to upload a high volume of small files (less or equal 100 Kb) to object storage. To enhance the transfer speed, both Aspera server and client are required to have Aspera v.3.4.6 or higher and be configured based on the following instructions. Using the recommended configuration documented below, our benchmark throughput of uploading 50,000 files averaging 100 KB in size from Windows client to object storage is 234 Megabits per second.

Recommended Configuration

I. Server Side

 To check your existing version on your server, you can run

# ascp -A

If your ascp is below v.3.4.6, please upgrade. You can download the latest software from our website. 

Once your server is running on v.3.4.6 or higher, perform the following configuration:

A) Disable Pre-calculate job size either via aspera.conf or through Aspera Console

1) via aspera.conf

In /opt/aspera/etc/aspera.conf,insert <pre_calculate_job_size>no</pre_calculate_job_size> in the <file_system> tab.

        <file_system>
            ...
            ...            <pre_calculate_job_size>no</pre_calculate_job_size>
        </file_system>
NOTE: This can be accomplished with the following command, run as root:
# asconfigurator -x "set_node_data;pre_calculate_job_size,no"

2) Via Aspera Console

Log on to Aspera Console as admin, go to Nodes->edit->Configuration->Advanced File Handling and set Pre-Calculate Job Size to "no".

pre_cal_no.jpg

B) Run the trap setup script and modify asperatrapd

 # /opt/aspera/bin/astrap-config.sh enable

Then, modify the trap configuration.

For On Demand versions 3.5.4 and earlier, do the following:

# vi /etc/init.d/asperatrapd

In the trap configuration, find the line  # disable checks done on open-write for small files and uncomment the following:

#PROPS="$PROPS -Daspera.session.upload.start-check-small-file.enabled=false"

For On Demand versions 3.6.0 and later, do the following:

# vi /opt/aspera/etc/trapd/trap.properties

In the trap configuration, find the following line:

#aspera.session.upload.start-check-small-file.enabled=true
Uncomment it and change the value to false:
aspera.session.upload.start-check-small-file.enabled=false
NOTE: When you make this change, the AsperaTrap service will no longer check if there is a directory with the same name that you are try to upload in object storage. Thus, if you upload file named "foo" and if there is already a directory called "foo" in your object storage container, you will end up with a file "foo" AND a directory "foo".

Restart trapd

# /etc/init.d/asperatrapd restart

II. Client Side

v.3.4.6 or higher is required on the client side. To check your existing version, from a command prompt or terminal, you can run

# ascp -A

If you are running on an older version of Aspera, please upgrade. You can download the latest software from our website.

Once your client is running on v.3.4.6 or higher, disable resume.

Disable Resume

Go to Connections > Edit > File Handling, and uncheck Resume incomplete files then click OK.

no_resume.jpg

If you must use resume, use the "-k1" option, "Compare file attributes".

"-k1" option for resume

Go to Connections > Edit > File Handling, change the Resume setting from the defaults, Compare sparse file checksums and Overwrite if the source is different

default_setting.jpg

to Compare file attributes and Always overwrite.

small_file_setting.jpg 

 Click OK.

 In addition, if you have a large volume of data to transfer, you can disable calculation of source files size before transferring to speed up the starting of transfer session. To do so, go to Connection > Edit > Transfer and check Disable Calculation of source files size before transferring.

disable_cal.jpg

 

Once you have the recommended configuration done on both the server and client, you should see transfer speed enhanced.  Besides Aspera GUI, you can also utilize the command line to upload files to object storage. 

For example,

# ascp -l 300000 -Q --mode=send --user=asp1 --host=54.235.47.47  50K_100KB  /100/SOL-Comp-SmallFiles/FromWin/Set2
50K_100KBFile9998.tmp                         100%  100KB  247Mb/s    02:54
50K_100KBFile9999.tmp                         100%  100KB  247Mb/s    02:54
FileCreate.bat                                100%   76    247Mb/s    02:54
Completed: 5000000K bytes transferred in 175 seconds
 (233876K bits/sec), in 50001 files, 1 directory.
Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.
Powered by Zendesk