On 04/15/2016 09:21 AM, Pat Riehecky wrote: > On 04/14/2016 08:20 PM, Steven Haigh wrote: >> On 2016-04-15 02:45, Pat Riehecky wrote: >>> On 04/14/2016 11:32 AM, Orion Poplawski wrote: >>>> On 04/14/2016 08:54 AM, Pat Riehecky wrote: >>>>> On 04/14/2016 09:35 AM, Orion Poplawski wrote: >>>>>> On 03/30/2016 10:44 AM, Pat Riehecky wrote: >>>>>>> On 03/30/2016 11:39 AM, Orion Poplawski wrote: >>>>>>>> On 03/29/2016 08:55 AM, Orion Poplawski wrote: >>>>>>>>> I get messages like this fairly frequently on our SL7 servers: >>>>>>>>> >>>>>>>>> /etc/cron.daily/0yum-daily.cron: >>>>>>>>> >>>>>>>>> Not using downloaded repomd.xml because it is older than what >>>>>>>>> we have: >>>>>>>>> Current : Fri Mar 25 08:54:39 2016 >>>>>>>>> Downloaded: Fri Mar 25 08:54:20 2016 >>>>>>>>> >>>>>>>>> Today, this is from the sl-security repo: >>>>>>>>> >>>>>>>>> -rw-r--r--. 1 root root 3059 Mar 25 08:54 >>>>>>>>> /var/cache/yum/x86_64/7/sl-security/repomd.xml >>>>>>>>> >>>>>>>>> We use a mirroring scheme that is based on time-stamps which >>>>>>>>> may be >>>>>>>>> aggravating this locally, and are mirroring from >>>>>>>>> http://ftp1.scientificlinux.org/linux/scientific >>>>>>>>> >>>>>>>>> However it seems that the time-stamps of these files should >>>>>>>>> never go >>>>>>>>> backwards. Is there something going on in the SL staging >>>>>>>>> process causing >>>>>>>>> this? >>>>>>>> A little more information. Apparently the <revision> tag in the >>>>>>>> repomd.xml is >>>>>>>> treated as the timestamp of the file. Currently for the >>>>>>>> currently downloaded >>>>>>>> sl-security the revision is: >>>>>>>> >>>>>>>> $ grep revision /var/cache/yum/x86_64/7/sl-security/repomd.xml >>>>>>>> <revision>1458917662</revision> >>>>>>>> $ ls -l /var/cache/yum/x86_64/7/sl-security/repomd.xml >>>>>>>> -rw-r--r--. 1 root root 3059 Mar 25 08:54 >>>>>>>> /var/cache/yum/x86_64/7/sl-security/repomd.xml >>>>>>>> $ date --date=@1458917662 >>>>>>>> Fri Mar 25 08:54:22 MDT 2016 >>>>>>>> $ stat --printf '%y\n' >>>>>>>> /var/cache/yum/x86_64/7/sl-security/repomd.xml >>>>>>>> 2016-03-25 08:54:40.000000000 -0600 >>>>>>>> >>>>>>>> Now I don't understand how the cached copy ended up with a >>>>>>>> slightly newer >>>>>>>> timestamp - possibly due to some munging by yum after download. >>>>>>>> >>>>>>>> For the current version on my mirror: >>>>>>>> >>>>>>>> # ls -l 7x/x86_64/updates/security/repodata/repomd.xml >>>>>>>> -rw-r--r--. 1 apache apache 2965 Mar 29 07:46 >>>>>>>> 7x/x86_64/updates/security/repodata/repomd.xml >>>>>>>> # grep revision 7x/x86_64/updates/security/repodata/repomd.xml >>>>>>>> <revision>1458917662</revision> >>>>>>>> >>>>>>>> So despite being newer and different file, it has the same >>>>>>>> "revision" as the >>>>>>>> older one. >>>>>>>> >>>>>>>> Perhaps you are manually setting --revision in createrepo? By >>>>>>>> default it >>>>>>>> sets >>>>>>>> revision to the unix timestamp at the start of the run. >>>>>>>> >>>>>>> I'll alter the publication tools to add this value >>>>>>> >>>>>>> Pat >>>>>> Seems to have happened again: >>>>>> >>>>>> # /etc/cron.daily/0yum-daily.cron >>>>>> Not using downloaded repomd.xml because it is older than what we >>>>>> have: >>>>>> Current : Wed Apr 13 09:59:29 2016 >>>>>> Downloaded: Wed Apr 13 09:59:28 2016 >>>>>> >>>>>> -rw-r--r--. 1 root root 3059 Apr 13 09:59 >>>>>> /var/cache/yum/x86_64/7/sl-security/repomd.xml >>>>>> >>>>> http://ftp.scientificlinux.org/linux/scientific/7x/x86_64/updates/security/repodata/repomd.xml >>>>> >>>>> >>>>> >>>>> Contains the revision field you requested. >>>>> >>>>> Data from the publication filesystem: >>>>> $ pwd >>>>> /linux/scientific/7x/x86_64/updates/security/repodata >>>>> $ stat repomd.xml >>>>> File: ‘repomd.xml’ >>>>> Size: 2965 Blocks: 64 IO Block: 32768 regular >>>>> file >>>>> Device: 2ah/42d Inode: 905695246 Links: 1 >>>>> Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) >>>>> Context: system_u:object_r:nfs_t:s0 >>>>> Access: 2016-04-14 07:01:06.288136000 -0500 >>>>> Modify: 2016-04-13 11:06:36.036051000 -0500 >>>>> Change: 2016-04-13 11:06:36.036051000 -0500 >>>>> Birth: - >>>>> $ >>>>> >>>>> Pat >>>> Okay, it appears that I misread some of the code originally. It >>>> appears that >>>> the "timestamp" of the repomd file is the latest timestamp of all >>>> of the >>>> component data entries. >>>> >>>> I've attached the old and new repomd.xml files that I have. Both >>>> have the >>>> same revision, but the timestamps of the data files are different. >>>> The order >>>> and structure of the files are different as well so something >>>> reprocessed them. >>>> >>>> The latest timestamp in the old file is 1460563169 from >>>> updateinfo. The >>>> latest timestamp in the new file is 1460563168 from primary_db. >>>> updateinto is >>>> not in the old repomd.xml file. >>>> >>>> >>> Hmmmmm..... >>> >>> We use modifyrepo to inject the updateinfo data based on the packages >>> within the metadata. >>> >>> I'll see about adjusting the process there. >> >> Is there any reason modifyrepo is used instead of just createrepo? I >> understand it may take longer to do a full (re)create, but I've never >> run into this problem when I've done repos. >> >> My workflow is rather simple, but I do: >> >> #!/bin/bash >> if [ -z $1 ]; then >> echo "Usage: $0 <reponame>" >> exit 1 >> fi >> >> createrepo --outputdir=/path/to/repo/$1/x86_64/ /path/to/repo/$1/x86_64/ >> >> rsync --delete-after --progress -a /path/to/$1/ >> livesite:/path/to/repo/$1/ >> exit 0 >> >> You could optionally use the --update parameter to update an existing >> repo - which may be faster. >> > > The createrepo command does not insert the updateinfo metadata even > when present within the repodata directory. Like the group > information, it comes from an external data source, but unlike group > info there isn't a switch to load it in. > > I can look into switching to createrepo_c which has the relevant > switches, but that will take a bit as we'll need to modify the > publication workflow. > > I'd like to try one more possible workaround before investing the time > there. Two createrepo commands feels a bit strange and like the kind > of thing I'd optimize out down the road if this isn't sufficiently > well documented. > > Pat I've rebuild the 7 security repos with my workaround in place. Pat