High Performance synchronization of large number of small files

Summary

This article documents performance options for using async with large numbers of small files.  The testing and demonstrations were done on Windows systems, but all of the settings do apply in Linux.

Details

Here is an example command.  Details are provided below.

"C:\Program Files\Aspera\Enterprise Server\bin\async.exe" -L C:\Logs:200 -k none -q -N MIGRATE -c none --pending-max=10000 --preserve-acls=native --transfer-threads=4 -R C:\Logs:200 -d C:\data\ -r aspera@192.168.35.117:"C:\data\" -w aspera -K PUSH -a fair -l 500m -x 

Logging 

Although not necessary, it can come in handy to create a separate log file for your sync job.  The -L is for local, and -R for remote.  The addition of ':200' will enable the logs to grow to 200 MB before being rotated. 

-L C:\Logs:200   (local log folder - goes at the beginning of the command)
-R C:\Logs:200   (remote log folder - remote log configuration must come after -N)

Resume

The resume option is not useful for small files. 

-k none

Encryption

If you already have a secure connection between source and destination, you should consider turning off encryption to get a boost in performance.

-c none

Pending files

It is helpful to increase the number of files pending transfer.  With small files, you can easily be blocked at the default of 2,000.

--pending-max=10000

Preserving Windows Permissions

Data center migrations often require the preservation of permissions and ACLs.

--preserve-acls=native

Multiple transfer threads

Since ascp is single threaded, you can often get improved performance by using multiple ascp processes to move the data.  In my example, we set the limit to 4.  This value should match the number of CPU cores.

--transfer-threads=4

 

 

 

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.
Powered by Zendesk