Hi all:

I'm running a local mirror right now, using a version of Nico's script. Although having a bunch of machines update against the SL repos directly instead of locally was an issue for bandwidth reasons, the bigger win for me is to be able to manually control syncing on my mirror such that my o/s patching is always working against something stable. Woe unto you that assumes their yum update on 2 different machines is the same if you don't have a mirror you control, or you don't check your /etc/yum.log.



Tim  Kanuka

Canadian Light Source Inc.



-----Original Message-----

From: [log in to unmask] [mailto:[log in to unmask]] On Behalf Of Nico Kadel-Garcia

Sent: November 24, 2015 06:01

To: Benjamin Lefoul <[log in to unmask]>

Cc: [log in to unmask]

Subject: Re: Is a mirror what I want?



On Tue, Nov 24, 2015 at 4:41 AM, Benjamin Lefoul <[log in to unmask]> wrote:

> Hi all,

>

> I thought it could be a good idea to have a full local copy of SL on some of our systems. Is a mirror what I want?



Depends. If you've only one SL machine, and you're not constantly rebuilding it or using something like "mock" that rebuilds chroot cages frequently, or using a "chef" setup that keeps clearing your local yum cache, you may not want to spend the work and the bandwidth and the disk space to maintain  a mirror.



If you're like me, and you build big stacks of customized software using "mock", and you need to keep rebuilding your chroot cage after rebuilding every single package because other packages may need it, well then! A local mirror can easily cut your build time in half, and leave your bandwidth for your roommates to use, and the upstream repositories will appreciate your not hammering them so much.



If you need them, I published scripts for this way back when, at :https://github.com/nkadel/nkadel-rsync-scripts. They're mostly based on "rsync".  My different scripts included Scientific Lunux, EPEL, CentOS, and even some reposync tools for making mirrors of RHEL published yum repositories. Those were invaluable for me to be able to switch from direct RHEL to Scientific Linux to CentOS compilation for "mock", and to maintain local mirrors in 100 machine environments where external bandwidth for chef based systems, which kept clearing and surveying yum based package lists, got overwhelming.



If disk space is tight for your mirror, my scripts included hooks to exclude some bulky components. The libreoffice langpacks, for example, are quite bulky and tend to churn, and if you've got a local mirror you probably don't need DVD images. And the honking huge game data files on EPEL are *not* needed for most local mirrors, though having a local EPEL: mirror as well as an SL mirror can also profoundly improve external bandwidth use and improve yum operations.



> For the same problem, Debian advises (https://www.debian.org/mirror/ftpmirror) : "Is a mirror the right choice? Sometimes people mistakenly start a mirror, when they actually want to run a caching proxy, such as apt-cacher-ng."

> But that is a debian tool, and I am not even sure I understand how it is different from mirroring...



Debian "apt" has what I think is a more sensible approach to handling the metadata about individual packages, I think. Each package has its own metadata published apart from that package in an individual file, and it's the client system's problem to pull all those and assemble an update list. Someone at Yellowdong, a long time ago, writing the "Yellowdog Update Manager", or "yum", decided that all the packages should have their metadata stored in a single shared database. This is the yum repodata, and it's gotten out of hand since then. The change, addition, or deletion of a single RPM package means that *all* the repodata needs to be rebuilt, and *all* of it transferred.



*That* is often the data that winds up getting re-transferred, and re-transferred, and re-transferred, and sucking your bandwidth and churning your local disk. For most of us, we can avoid this by editing "/etc/yum.conf" and setting "metadata_expire" to something sensible, like 720m.



I've looked at the new "dnf" system for RHEL 7. and in Fedora, and I'm afraid it doesn't actually solve any of the real problems. The ability to assemble "diff" based updates on top of existing packages is clever, but doesn't actually help normal operation and expands the size of upstream mirrors pointlessly.



> Do we have something similar, maybe based on yum / dnf?

>

> In fact, I would welcome any comment on how to set a full local copy of SL7 and in which respective context...

>

> Thanks!

>

> Benjamin Lefoul

> nWISE AB