This article is for customers who need to upload a high volume of small files (less or equal 100 Kb) to object storage. To enhance the transfer speed, both Aspera server and client are required to have Aspera v.3.4.6 or higher and be configured based on the following instructions. Using the recommended configuration documented below, our benchmark throughput of uploading 50,000 files averaging 100 KB in size from Windows client to object storage is 234 Megabits per second.
I. Server Side
To check your existing version on your server, you can run
# ascp -A
If your ascp is below v.3.4.6, please upgrade. You can download the latest software from our website.
Once your server is running on v.3.4.6 or higher, perform the following configuration:
A) Disable Pre-calculate job size either via aspera.conf or through Aspera Console
1) via aspera.conf
<file_system> ... ... <pre_calculate_job_size>no</pre_calculate_job_size> </file_system>
# asconfigurator -x "set_node_data;pre_calculate_job_size,no"
2) Via Aspera Console
Log on to Aspera Console as admin, go to Nodes->edit->Configuration->Advanced File Handling and set Pre-Calculate Job Size to "no".
B) Run the trap setup script and modify
# /opt/aspera/bin/astrap-config.sh enable
Then, modify the trap configuration.
For On Demand versions 3.5.4 and earlier, do the following:
# vi /etc/init.d/asperatrapd
In the trap configuration, find the line
# disable checks done on open-write for small files and uncomment the following:
For On Demand versions 3.6.0 and later, do the following:
# vi /opt/aspera/etc/trapd/trap.properties
In the trap configuration, find the following line:
# /etc/init.d/asperatrapd restart
II. Client Side
v.3.4.6 or higher is required on the client side. To check your existing version, from a command prompt or terminal, you can run
# ascp -A
If you are running on an older version of Aspera, please upgrade. You can download the latest software from our website.
Once your client is running on v.3.4.6 or higher, disable resume.
Go to Connections > Edit > File Handling, and uncheck Resume incomplete files then click OK.
If you must use resume, use the "-k1" option, "Compare file attributes".
"-k1" option for resume
Go to Connections > Edit > File Handling, change the Resume setting from the defaults, Compare sparse file checksums and Overwrite if the source is different
to Compare file attributes and Always overwrite.
In addition, if you have a large volume of data to transfer, you can disable calculation of source files size before transferring to speed up the starting of transfer session. To do so, go to Connection > Edit > Transfer and check Disable Calculation of source files size before transferring.
Once you have the recommended configuration done on both the server and client, you should see transfer speed enhanced. Besides Aspera GUI, you can also utilize the command line to upload files to object storage.
# ascp -l 300000 -Q --mode=send --user=asp1 --host=18.104.22.168 50K_100KB /100/SOL-Comp-SmallFiles/FromWin/Set2
50K_100KBFile9998.tmp 100% 100KB 247Mb/s 02:5450K_100KBFile9999.tmp 100% 100KB 247Mb/s 02:54FileCreate.bat 100% 76 247Mb/s 02:54Completed: 5000000K bytes transferred in 175 seconds(233876K bits/sec), in 50001 files, 1 directory.