Improving NFS Performance on the LAN

by John Heaton

Why use NFS

Aspera is used to overcome the transfer problem that exists on the Wide Area Network (WAN). In many instances, the backend storage for Aspera nodes is a Network Attached Storage (NAS) devices like a NetApp Filer or Isilon Clustered solution. The NAS is a LAN solution (for more info on storage alternatives, please see our intro to storage) that provides flexible, expandable high performance storage. While several protocols are available to access NAS storage, the focus of this article is NFS).

NFS has a long history in providing remote file access; it is currently at version 4 of the specification and currently used in primarily Unix/Linux deployments. The design of the protocol is highly efficient for data access, which is different than CIFS which was designed for workgroup access to files. For this reason, NFS should be the preferred NAS access protocol for high throughput applications.

It should be noted that the performance settings shown below were done on a NetApp FAS6070. The storage performance capabilities will change with each platform. Also, these tests were done without the use of encryption on the fasp™ transfer. Turning encryption on will affect the possible throughput by using up more of the processor than could otherwise go to file transfer.

Hardware and Network Setup

To achieve the best performance possible, it is advisable to do the following:

  • Separate traffic onto different Network Interface Cards (NIC): one for WAN traffic, and one for LAN storage. This will become important later on, but practically speaking, most servers now have two or more NICs. Additionally, high performance IP storage is normally segregated onto another, non-routable network, much like a SAN fabric. By having this separation, the hardware can be used for the sole purpose of WAN or LAN data, making optimal use of the hardware resources in preventing a bottleneck.
  • Universal access to the storage system may be one goal of using a NAS, but if limited access is allowable, then having a storage network provides flexibility to tune storage performance up. Smaller, workgroup LANs may serve double or triple duty, so they are optimized to be general purpose. To get this general purpose functionality, network administrators may impose QoS (Quality of Service) or other techniques that can limit NAS performance. A storage LAN would allow for the use of Jumbo Frames (reducing packet overhead) and switches with larger backplanes (that can support sustained throughputs), among other things.

Setting up Linux

Getting the latest kernel

This is a general comment. Using the latest kernel typically means optimizations to the IP and NFS stacks are available.

Setting kernel variables

In /etc/sysctl.conf set the File System to be more aggressive in flushing data to disk. This is done by changing or adding the following to the values suggested below:

vm.dirty_ratio = 0 
vm.dirty_background_ratio = 0

Increase maximum number of incoming packets queued up for upper-layer processing:

net.core.netdev_max_backlog = 30000

Adjust the NIC (assuming eth0) to have a large queue. Useful on 10Gig cards:

$ ifconfig eth0 txqueuelen 10000

To adjust the NIC permanently so it persists after reboots, open the following file:

  • /etc/rc.local

Add the following line:

/sbin/ifconfig eth0 txqueuelen 10000

Setup the mount options

Depending on the type of data, it may be good to disableatime, or put another way, mount withnoatime.

Set CPU Affinity

Check the IRQ or interrupt load per processor. If a processor is getting an uneven load of the work for processing interrupts,irqbalance may need to be disabled or manual steps take to rebalance interrupts.

Use Separate Network Interfaces

Separate WAN and LAN (storage) traffic to ensure that no contention is caused by either set of traffic with respect to one another.

Next Steps

This guide will be updated with details when relevant. It is intended to be a guide for people already versed in Linux tuning.

Powered by Zendesk