On Mon, 11 Apr 2011, Stephen John Smoogen wrote:

> On Mon, Apr 11, 2011 at 05:08, Federico Alves <[log in to unmask]> wrote:
>> The limitation is not my hardware. The servers are both Dell R900 with SAS
>> disk arrays. Also, from a Windows virtual machine, inside the same server,
>> I get around 400 MB speed using FTP transfer, windows to windows. There
>> must be a different way to do this from Linux.The files are sparse files,
>> and I need to keep them that way, that's why I use rsync.
>
> Well transferring sparse files is going to be slow and it could be
> hardware (unless you are somehow testing with windows of copying
> sparse files over). rsync is having to see what real bits are there
> and what is fluff so it is going to be CPU and disk intensive.

Just to add to that the rsync --sparse option makes the *transfer* of 
sparse data efficient, but unless it has an efficient way to detect the 
unallocted areas it will still need to read through the file looking for 
blank bits.

Even having to do that the --sparse will save effort since at least it 
won't need to *transfer* the large blocks of emptyness to the far end...

Newer versions of rsync have an option to specify the sparse-blocksize to 
use, which apparently can vastly improve things on some high-end hardware.

At least some file-systems allow the detection of whch blocks are actually 
allocated and in theory rsync could use those but it wouldn't be portable. 
e.g. for Solaris' zfs Sun added SEEK_HOLE and SEEK_DATA to their lseek 
options which is probably better than the various other hacks that have 
been implemented over the years but I don't think it is widely implemented 
outside of Solaris...  e.g. see:

   http://lwn.net/Articles/260795/

which has a description of SEEK_HOLE/DATA and the FIEMAP ioctl() which can 
already be used on some (I don't know which) linux fs...  I doubt that sl5 
will be likely to support either, and I wouldn't like to guess about sl6.

  -- Jon