On Fri, Jun 3, 2016 at 9:38 AM, Alec T. Habig <[log in to unmask]> wrote:

Hmm.  I've (independently from the original poster) got the 8k sizes:

Try running "mount -o nfsvers=3 myserver:/mydir /mnt" by hand between two SL6.7 systems. I just did, and it gives me proto=tcp and rsize/wsize of 1024k.

I vaguely recall seeing firewall oddities cause NFS mounts to fall back to UDP with an 8k rsize/wsize. So check the "proto=tcp" carefully if you see something else.
 
with the network set up for jumbo frames (MTU=9000) the nfs server then
chunks out network packets that get through the switches with minimal
overhead.  Making this change from the default (1k, MTU=1500) made a
huge throughput difference at the time we implemented it: which was a
number of years ago, so certainly the world has changed since then.

How do 1MB sized nfs chunks interact with the networking?

The same way any other application over TCP interacts with the networking... TCP will break the (large) NFS requests and responses into MTU-sized segments on send and then reassemble them on receive. The reason to let TCP do this (instead of trying to match the application payloads to the MTU) is that modern systems offload both the breakup and the reassembly to the network hardware. Try searching for "TCP Segmentation Offload" and "Large Receive Offload" for details. Note that modern Linux uses different names for these things and I can't keep up, but the principle has not changed.

Last time I experimented, these offload engines largely obviated the advantages of jumbo frames. Jumbo frames do make a huge difference with older hardware/software and with UDP, though.

These offload engines are why it is hard to increase NFS throughput via channel bonding, because bonded channels generally have to disable these engines.

Incidentally, I typically see 900MB/sec sustained NFS reads and writes while moving half-terabyte files. (My application is somewhat specialized.) Of course this requires a fast disk subsystem and 10GbE interconnect.

 - Pat