SCIENTIFIC-LINUX-USERS Archives

September 2011

SCIENTIFIC-LINUX-USERS@LISTSERV.FNAL.GOV

Options: Use Monospaced Font
Show HTML Part by Default
Show All Mail Headers

Message: [<< First] [< Prev] [Next >] [Last >>]
Topic: [<< First] [< Prev] [Next >] [Last >>]
Author: [<< First] [< Prev] [Next >] [Last >>]

Print Reply
Subject:
From:
Andrew Z <[log in to unmask]>
Reply To:
Date:
Wed, 28 Sep 2011 21:27:04 -0400
Content-Type:
multipart/alternative
Parts/Attachments:
text/plain (1352 bytes) , text/html (1995 bytes)
Pablo,
I'm not sure if my experience is relevant for your case, but...
as a part of etl job, I needed to process rather large files - a few millions records, with total count of files in low 100. I dynamically grouped them and feed each group to scp job. Since those were plain txt files with tons spaces, I did add compression.
disclaimer: that was done on 16 cpu boxes with 64gb memory, so I was all but cpu bound. I did got screamed by sa for saturating the network. But it was easy to adjust by decreasing amount of groups.
Hope this helps.
Andrew
-- 
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

Pablo Cavero <[log in to unmask]> wrote:

Hi,

I want to know if exist any tips to have a faster scp transfer. In the Client, or in the Server Config.

I'm testing use the standar SCP and the PSCP, a command extra from the Puty apps.

Home page of Putty:
http://www.chiark.greenend.org.uk/~sgtatham/putty/

Link to Download the File, to make and compiling the new commands:
http://the.earth.li/~sgtatham/putty/latest/putty-0.61.tar.gz

Well, I see a few best performance in the standar scp, but this command, don't support run in a batch, add the user and password like parameter.
And generate and share the RSA public kay is too much work, when some one have to many clients.

Any Idea??

-- 
Pablo Cavero
System Engineer
+569 8920 9509



ATOM RSS1 RSS2