High Performance synchronization of large number of small files


This article documents performance options for using async with large numbers of small files.  The testing and demonstrations were done on Windows systems, but all of the settings do apply in Linux.


Here is an example command.  Details are provided below.

"C:\Program Files\Aspera\Enterprise Server\bin\async.exe" -L C:\Logs:200 -k none -q -N MIGRATE -c none --pending-max=10000 --preserve-acls=native --transfer-threads=4 -R C:\Logs:200 -d C:\data\ -r aspera@"C:\data\" -w aspera -K PUSH -a fair -l 500m -x 


Although not necessary, it can come in handy to create a separate log file for your sync job.  The -L is for local, and -R for remote.  The addition of ':200' will enable the logs to grow to 200 MB before being rotated. 

-L C:\Logs:200   (local log folder - goes at the beginning of the command)
-R C:\Logs:200   (remote log folder - remote log configuration must come after -N)


The resume option is not useful for small files. 

-k none


If you already have a secure connection between source and destination, you should consider turning off encryption to get a boost in performance.

-c none

Pending files

It is helpful to increase the number of files pending transfer.  With small files, you can easily be blocked at the default of 2,000.


Preserving Windows Permissions

Data center migrations often require the preservation of permissions and ACLs.


Multiple transfer threads

Since ascp is single threaded, you can often get improved performance by using multiple ascp processes to move the data.  In my example, we set the limit to 4.  This value should match the number of CPU cores.





Powered by Zendesk