Hi Troy,

this problem has become so frequent that I stopped reporting it, since I
was sure your storage experts are chasing it.

I usually run a dryrun rsync whenever I expect some change (it's not a
cron job). Lately, at least one out of four times the dryrun told me it
would delete some files it clearly shouldn't, typically with the first
character of the filename from some range in the alphabet, as in the
reports your received.

Every time I checked (maybe the first five times), using an ftp client 
confirmed what rsync said: the server didn't know about the files.

Given that an rsync dryrun takes about a minute, the problem must be 
present a pretty significant fraction of the day. I usually have to wait 
for O(30m) until the files "are back". Now we typically run this in the 
morning. I've oserved the around noon as well (all times GMT+1). Hence it 
may be due to some maintenace task you or your storage guys run at night.

It should be fairly simple to gain more statistics: just running an "ls
<release>/SL/RPMS/" (or /SL/ for SL5) every 15 minutes on the server,
for all stable releases, an comparing with the first result, should 
provide you with evidence that the problem is real and an idea of how 
serious it is, without any risk of false positives or significant load on 
the server.

And please cut Kai a little slack. He's in charge of mirroring the SL
tree here (not trivial lately due to the problem this is all about), and 
deciding which updates make it onto the systems and which don't yet, 
while I'm on leave. For example, right now he has to put the xen updates 
for 5.0 on hold. As he said, he's new to the SL business, but he's not a 
fool. 5.1 beta is integrated into our installation and configuration 
management under the 5.1 label, which is probably why he confused 
5rolling for 5.1.

Cheers,
 	Stephan

On Thu, 6 Dec 2007, Troy Dawson wrote:

> Hi Martin,
> Could you let me know what time this happened, and what machine is doing the 
> transfer.
> It's actually much easier for us to trace ftp transactions than rsync, so 
> maybe this will give us a glimpse into what is happening better than tracking 
> down DESY's rsync.
>
> Troy
>
> Bly, MJ (Martin) wrote:
>> I am seeing this too, on a long-standing mirror using lftp direct from
>> ftp.scientificlinux.org.  Clobbered some installations I was doing of
>> our copy on Sunday - we lost some of the SL4.4 i386 base stuff, mostly
>> files starting 'o'  (openssl, openoffice, openssh, ... list on request).
>> A re-sync of the mirror brought the files back.
>>
>>         Martin.
>> --
>> Martin Bly
>> RAL Tier1 Fabric Team
>> 
>>> -----Original Message-----
>>> From: [log in to unmask]
>>> [mailto:[log in to unmask]] On
>>> Behalf Of Kai Leffhalm
>>> Sent: 06 December 2007 11:38
>>> To: [log in to unmask]
>>> Subject: Problem with Mirroring
>>> 
>>> Hi,
>>> I am new here and have one problem when mirroring the repositories:
>>>   sometimes the packages disappear and reappear after some
>>> time. I was told that this is a problem
>>> with the fileserver. Now there are some old packages, which
>>> seems to be deleted on purpose and some
>>> seems to be reorganized.
>>> 
>>> Is there any possibility to see, which files would be deleted
>>> on purpose and which are just
>>> temporarily deleted?
>>> 
>>> Cheers
>>> Kai Leffhalm
>>> 
>>> Examples:
>>> SL5.1: (Files are moved to different directory)
>>> x86_64/SL/firefox-1.5.0.12-7.el5.i386.rpm
>>> x86_64/SL/firefox-1.5.0.12-7.el5.x86_64.rpm
>>> x86_64/SL/firefox-devel-1.5.0.12-7.el5.i386.rpm
>>> x86_64/SL/firefox-devel-1.5.0.12-7.el5.x86_64.rpm
>>> deleting x86_64/updates/security/firefox-1.5.0.12-7.el5.x86_64.rpm
>>> deleting x86_64/updates/security/firefox-1.5.0.12-7.el5.i386.rpm
>>> deleting
>>> x86_64/updates/security/firefox-devel-1.5.0.12-7.el5.x86_64.rpm
>>> deleting x86_64/updates/security/firefox-devel-1.5.0.12-7.el5.i386.rpm
>>> 
>>> 
>>> SL4.4: (Old and current packages are deleted)
>>> deleting i386/errata/SL/RPMS/postgresql-libs-7.4.17-1.RHEL4.1.i386.rpm
>>> deleting i386/errata/SL/RPMS/postgresql-jdbc-7.4.17-1.RHEL4.1.i386.rpm
>>> deleting i386/errata/SL/RPMS/openssh-server-3.9p1-8.RHEL4.20.i386.rpm
>>> deleting i386/errata/SL/RPMS/openssh-clients-3.9p1-8.RHEL4.20.i386.rpm
>>> 
>
>
>

-- 
Stephan Wiesand
   DESY - DV -
   Platanenallee 6
   15738 Zeuthen, Germany