SCIENTIFIC-LINUX-USERS Archives

April 2011

SCIENTIFIC-LINUX-USERS@LISTSERV.FNAL.GOV

Options: Use Monospaced Font
Show Text Part by Default
Show All Mail Headers

Message: [<< First] [< Prev] [Next >] [Last >>]
Topic: [<< First] [< Prev] [Next >] [Last >>]
Author: [<< First] [< Prev] [Next >] [Last >>]

Print Reply
Subject:
From:
Florian Philipp <[log in to unmask]>
Reply To:
Florian Philipp <[log in to unmask]>
Date:
Mon, 11 Apr 2011 10:23:30 +0200
Content-Type:
multipart/signed
Parts/Attachments:
text/plain (1935 bytes) , signature.asc (268 bytes)
Am 11.04.2011 13:08, schrieb Federico Alves:
> 
> On 4/10/11 11:35 PM, "Larry Brower" <[log in to unmask]> wrote:
> 
> On 04/11/2011 05:59 AM, Federico Alves wrote:
>>>> I am using rsync to send almost 1 TB of sparse files across the LAN to
>>>> another identical Linux box. If I fire only 1 command, I get about 22 Mb
>>>> of speed, but if I fire 6 commands in parallel, from different SSH
>>>> connections, the speed is divided by 6: very, very slow.
>>>>
>>>> My command is
>>>>  rsync -S --progress sparsefile jephe@server:newsparsefile
>>>>
>>>> The LAN is 1GB and both machines are Scientific Linux 6.0.
>>>> Is there any way to do this that does not have a bottleneck?
>>>> Federico
> 
> 
> 
> This could be a limitation of your disks speed in the servers and not
> rsync itself
>>
>
> The limitation is not my hardware. The servers are both Dell R900
> with SAS disk arrays. Also, from a Windows virtual machine, inside
> the same server, I get around 400 MB speed using FTP transfer,
> windows to windows. There must be a different way to do this from
> Linux.The files are sparse files, and I need to keep them that way,
> that's why I use rsync.

Maybe it helps to rule out SSH as the culprit. Can you try to set up a
real rsync-server on that machine (package rsync-server)? Take a look at
`man rsyncd.conf` for an example configuration. Here is a short
walkthrough on how to setup one:
http://www.jtanderson.org/linux/centos-5-rsync-server-setup/
(BTW: Why isn't there an init script and default config file in the
rsync package?)

Alternatively, you can replace ssh with rsh and call `rsync --rsh=rsh ...`

If the problem is really rsync, you can try to stream tar over ssh.
tar --create --sparse $fileA $fileB | ssh user@server tar --extract
--directory $target_dir --preserve-permissions --sparse

BTW: Please don't top-post. Put your reply below the message you are
quoting. This makes reading long threads easier.

Hope this helps,
Florian Philipp



ATOM RSS1 RSS2