Chaps,
I've only bounced against this the one time: we do twice daily mirror
runs starting 2am and noon (UK time, currently GMT, thus 8pm and 6am
FNAL time) and this was the only time in almost daily use of our own
mirrors to install/update systems.
If you think it might help diagnose the problem, I can try and
instrument our mirror runs to pin the times of each mirror.
Martin.
--
-----------------------------------
Martin Bly +44|0 1235 446981
RAL Tier1 Fabric Team Manager
-----------------------------------
> -----Original Message-----
> From: [log in to unmask]
> [mailto:[log in to unmask]] On
> Behalf Of Troy Dawson
> Sent: 06 December 2007 20:18
> To: [log in to unmask]
> Cc: Bly, MJ (Martin); [log in to unmask]
> Subject: Re: Problem with Mirroring
>
> Hi Stephan,
> Thank you for the explanation, especially about how frequent
> it is. To be honest, your reports were the only one's we'd
> gotten about this problem, so we weren't aware of how bad it
> is. I'm also glad that Martin spoke up, because if all the
> reports come from the same place it sort of seems like it's a
> possible communication between places problem.
>
> And if he isn't reading this, please send my apologies to
> Kai. I guess I did snap at him, and I'm sorry.
> Here at Fermilab I've had to explain to people at least 6
> times in the past week that we haven't yet released SLF 5.1
> and explain to them what that means.
> So I guess my temper has gotten a bit thin on that subject.
>
> Troy
>
> [log in to unmask] wrote:
> > Hi Troy,
> >
> > this problem has become so frequent that I stopped
> reporting it, since
> > I was sure your storage experts are chasing it.
> >
> > I usually run a dryrun rsync whenever I expect some change
> (it's not a
> > cron job). Lately, at least one out of four times the
> dryrun told me
> > it would delete some files it clearly shouldn't, typically with the
> > first character of the filename from some range in the
> alphabet, as in
> > the reports your received.
> >
> > Every time I checked (maybe the first five times), using an
> ftp client
> > confirmed what rsync said: the server didn't know about the files.
> >
> > Given that an rsync dryrun takes about a minute, the
> problem must be
> > present a pretty significant fraction of the day. I usually have to
> > wait for O(30m) until the files "are back". Now we
> typically run this
> > in the morning. I've oserved the around noon as well (all times
> > GMT+1). Hence it may be due to some maintenace task you or
> your storage guys run at night.
> >
> > It should be fairly simple to gain more statistics: just running an
> > "ls <release>/SL/RPMS/" (or /SL/ for SL5) every 15 minutes on the
> > server, for all stable releases, an comparing with the
> first result,
> > should provide you with evidence that the problem is real
> and an idea
> > of how serious it is, without any risk of false positives or
> > significant load on the server.
> >
> > And please cut Kai a little slack. He's in charge of
> mirroring the SL
> > tree here (not trivial lately due to the problem this is
> all about),
> > and deciding which updates make it onto the systems and which don't
> > yet, while I'm on leave. For example, right now he has to
> put the xen
> > updates for 5.0 on hold. As he said, he's new to the SL
> business, but
> > he's not a fool. 5.1 beta is integrated into our installation and
> > configuration management under the 5.1 label, which is
> probably why he
> > confused 5rolling for 5.1.
> >
> > Cheers,
> > Stephan
> >
> > On Thu, 6 Dec 2007, Troy Dawson wrote:
> >
> >> Hi Martin,
> >> Could you let me know what time this happened, and what machine is
> >> doing the transfer.
> >> It's actually much easier for us to trace ftp transactions than
> >> rsync, so maybe this will give us a glimpse into what is happening
> >> better than tracking down DESY's rsync.
> >>
> >> Troy
> >>
> >> Bly, MJ (Martin) wrote:
> >>> I am seeing this too, on a long-standing mirror using lftp direct
> >>> from ftp.scientificlinux.org. Clobbered some installations I was
> >>> doing of our copy on Sunday - we lost some of the SL4.4 i386 base
> >>> stuff, mostly files starting 'o' (openssl, openoffice,
> openssh, ... list on request).
> >>> A re-sync of the mirror brought the files back.
> >>>
> >>> Martin.
> >>> --
> >>> Martin Bly
> >>> RAL Tier1 Fabric Team
> >>>
> >>>> -----Original Message-----
> >>>> From: [log in to unmask]
> >>>> [mailto:[log in to unmask]]
> On Behalf
> >>>> Of Kai Leffhalm
> >>>> Sent: 06 December 2007 11:38
> >>>> To: [log in to unmask]
> >>>> Subject: Problem with Mirroring
> >>>>
> >>>> Hi,
> >>>> I am new here and have one problem when mirroring the
> repositories:
> >>>> sometimes the packages disappear and reappear after
> some time. I
> >>>> was told that this is a problem with the fileserver. Now
> there are
> >>>> some old packages, which seems to be deleted on purpose and some
> >>>> seems to be reorganized.
> >>>>
> >>>> Is there any possibility to see, which files would be deleted on
> >>>> purpose and which are just temporarily deleted?
> >>>>
> >>>> Cheers
> >>>> Kai Leffhalm
> >>>>
> >>>> Examples:
> >>>> SL5.1: (Files are moved to different directory)
> >>>> x86_64/SL/firefox-1.5.0.12-7.el5.i386.rpm
> >>>> x86_64/SL/firefox-1.5.0.12-7.el5.x86_64.rpm
> >>>> x86_64/SL/firefox-devel-1.5.0.12-7.el5.i386.rpm
> >>>> x86_64/SL/firefox-devel-1.5.0.12-7.el5.x86_64.rpm
> >>>> deleting
> x86_64/updates/security/firefox-1.5.0.12-7.el5.x86_64.rpm
> >>>> deleting x86_64/updates/security/firefox-1.5.0.12-7.el5.i386.rpm
> >>>> deleting
> >>>> x86_64/updates/security/firefox-devel-1.5.0.12-7.el5.x86_64.rpm
> >>>> deleting
> >>>> x86_64/updates/security/firefox-devel-1.5.0.12-7.el5.i386.rpm
> >>>>
> >>>>
> >>>> SL4.4: (Old and current packages are deleted) deleting
> >>>> i386/errata/SL/RPMS/postgresql-libs-7.4.17-1.RHEL4.1.i386.rpm
> >>>> deleting
> >>>> i386/errata/SL/RPMS/postgresql-jdbc-7.4.17-1.RHEL4.1.i386.rpm
> >>>> deleting
> >>>> i386/errata/SL/RPMS/openssh-server-3.9p1-8.RHEL4.20.i386.rpm
> >>>> deleting
> >>>> i386/errata/SL/RPMS/openssh-clients-3.9p1-8.RHEL4.20.i386.rpm
> >>>>
> >>
> >>
> >
> > --
> > Stephan Wiesand
> > DESY - DV -
> > Platanenallee 6
> > 15738 Zeuthen, Germany
>
>
> --
> __________________________________________________
> Troy Dawson [log in to unmask] (630)840-6468 Fermilab
> ComputingDivision/LCSI/CSI DSS Group
> __________________________________________________
>
|
