Transfers of very small files

by Serban Simu

Overview

Aspera fasp is designed to transfer sets of small files very efficiently, without gaps between files. A file transfer of 1000 fles of 1MB each (for a total of 1GB) is typically just as efficient as transferring a single 1GB file.

However, there are certain system limitations on either the sending or the receiving machine that can cause transfers of small files to be slower.

The first consideration is the number of files that the system can open per second. Even at modest transfer rates (10 or 20 Mbps), transferring 10KB files requires opening several hundreds of files per second and typical systems don't allow it.

Secondly, if the transfer is set to resume using sparse checksums or full checksums, the additional processing reduces the number of files per second that the Aspera sender can process. Disabling "resume" or using resume based on "file system attributes" will alleviate this problem.

Note: the current version of Windows hot-folders transfer files in batches of 10s or 100s of files. That introduces a delay between each batch, so transferring 1000's of very small files using a hot folder is not as efficient as performing a manual transfer. This behavior will be eliminated in the next generation of the hot-folder implementation.

Here are some data points:

  • Transfers of 1-2 MB files can achieve 300-400 Mbps on a fast machine, but may be as slow as 100 Mbps on a Windows machine or when the files are stored on network-attached storage.
  • Transfers of 10KB files can achieve 10-20 Mbps but can be as slow as 2 or 3 Mbps on a Windows machine or when the files are stored on network-attached storage.

How to Improve

  • Disable "resume" or enable resume using "file system attributes" (avoid "sparse checksum" or "full checksum")
  • Place the files on local storage or directly attached storage (avoid network-attached storage because it introduces delay in opening the small files)
  • Transfer an entire directory in one transfer session instead of using hot folders (which might introduce delays in processing the files)
Have more questions? Submit a request

0 Comments

Article is closed for comments.
Powered by Zendesk