On 04/15/2016 09:21 AM, Pat Riehecky wrote:
> On 04/14/2016 08:20 PM, Steven Haigh wrote:
>> On 2016-04-15 02:45, Pat Riehecky wrote:
>>> On 04/14/2016 11:32 AM, Orion Poplawski wrote:
>>>> On 04/14/2016 08:54 AM, Pat Riehecky wrote:
>>>>> On 04/14/2016 09:35 AM, Orion Poplawski wrote:
>>>>>> On 03/30/2016 10:44 AM, Pat Riehecky wrote:
>>>>>>> On 03/30/2016 11:39 AM, Orion Poplawski wrote:
>>>>>>>> On 03/29/2016 08:55 AM, Orion Poplawski wrote:
>>>>>>>>> I get messages like this fairly frequently on our SL7 servers:
>>>>>>>>>
>>>>>>>>> /etc/cron.daily/0yum-daily.cron:
>>>>>>>>>
>>>>>>>>> Not using downloaded repomd.xml because it is older than what 
>>>>>>>>> we have:
>>>>>>>>>      Current   : Fri Mar 25 08:54:39 2016
>>>>>>>>>      Downloaded: Fri Mar 25 08:54:20 2016
>>>>>>>>>
>>>>>>>>> Today, this is from the sl-security repo:
>>>>>>>>>
>>>>>>>>> -rw-r--r--. 1 root root 3059 Mar 25 08:54
>>>>>>>>> /var/cache/yum/x86_64/7/sl-security/repomd.xml
>>>>>>>>>
>>>>>>>>> We use a mirroring scheme that is based on time-stamps which 
>>>>>>>>> may be
>>>>>>>>> aggravating this locally, and are mirroring from
>>>>>>>>> http://ftp1.scientificlinux.org/linux/scientific
>>>>>>>>>
>>>>>>>>> However it seems that the time-stamps of these files should 
>>>>>>>>> never go
>>>>>>>>> backwards.  Is there something going on in the SL staging 
>>>>>>>>> process causing
>>>>>>>>> this?
>>>>>>>> A little more information.  Apparently the <revision> tag in the
>>>>>>>> repomd.xml is
>>>>>>>> treated as the timestamp of the file.  Currently for the 
>>>>>>>> currently downloaded
>>>>>>>> sl-security the revision is:
>>>>>>>>
>>>>>>>> $ grep revision /var/cache/yum/x86_64/7/sl-security/repomd.xml
>>>>>>>>      <revision>1458917662</revision>
>>>>>>>> $ ls -l /var/cache/yum/x86_64/7/sl-security/repomd.xml
>>>>>>>> -rw-r--r--. 1 root root 3059 Mar 25 08:54
>>>>>>>> /var/cache/yum/x86_64/7/sl-security/repomd.xml
>>>>>>>> $ date --date=@1458917662
>>>>>>>> Fri Mar 25 08:54:22 MDT 2016
>>>>>>>> $ stat --printf '%y\n' 
>>>>>>>> /var/cache/yum/x86_64/7/sl-security/repomd.xml
>>>>>>>> 2016-03-25 08:54:40.000000000 -0600
>>>>>>>>
>>>>>>>> Now I don't understand how the cached copy ended up with a 
>>>>>>>> slightly newer
>>>>>>>> timestamp - possibly due to some munging by yum after download.
>>>>>>>>
>>>>>>>> For the current version on my mirror:
>>>>>>>>
>>>>>>>> # ls -l 7x/x86_64/updates/security/repodata/repomd.xml
>>>>>>>> -rw-r--r--. 1 apache apache 2965 Mar 29 07:46
>>>>>>>> 7x/x86_64/updates/security/repodata/repomd.xml
>>>>>>>> # grep revision 7x/x86_64/updates/security/repodata/repomd.xml
>>>>>>>>     <revision>1458917662</revision>
>>>>>>>>
>>>>>>>> So despite being newer and different file, it has the same 
>>>>>>>> "revision" as the
>>>>>>>> older one.
>>>>>>>>
>>>>>>>> Perhaps you are manually setting --revision in createrepo?  By 
>>>>>>>> default it
>>>>>>>> sets
>>>>>>>> revision to the unix timestamp at the start of the run.
>>>>>>>>
>>>>>>> I'll alter the publication tools to add this value
>>>>>>>
>>>>>>> Pat
>>>>>> Seems to have happened again:
>>>>>>
>>>>>> # /etc/cron.daily/0yum-daily.cron
>>>>>> Not using downloaded repomd.xml because it is older than what we 
>>>>>> have:
>>>>>>     Current   : Wed Apr 13 09:59:29 2016
>>>>>>     Downloaded: Wed Apr 13 09:59:28 2016
>>>>>>
>>>>>> -rw-r--r--. 1 root root 3059 Apr 13 09:59
>>>>>> /var/cache/yum/x86_64/7/sl-security/repomd.xml
>>>>>>
>>>>> http://ftp.scientificlinux.org/linux/scientific/7x/x86_64/updates/security/repodata/repomd.xml 
>>>>>
>>>>>
>>>>>
>>>>> Contains the revision field you requested.
>>>>>
>>>>> Data from the publication filesystem:
>>>>> $ pwd
>>>>> /linux/scientific/7x/x86_64/updates/security/repodata
>>>>> $ stat repomd.xml
>>>>>    File: ‘repomd.xml’
>>>>>    Size: 2965          Blocks: 64         IO Block: 32768 regular 
>>>>> file
>>>>> Device: 2ah/42d    Inode: 905695246   Links: 1
>>>>> Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: ( 0/ root)
>>>>> Context: system_u:object_r:nfs_t:s0
>>>>> Access: 2016-04-14 07:01:06.288136000 -0500
>>>>> Modify: 2016-04-13 11:06:36.036051000 -0500
>>>>> Change: 2016-04-13 11:06:36.036051000 -0500
>>>>>   Birth: -
>>>>> $
>>>>>
>>>>> Pat
>>>> Okay, it appears that I misread some of the code originally. It 
>>>> appears that
>>>> the "timestamp" of the repomd file is the latest timestamp of all 
>>>> of the
>>>> component data entries.
>>>>
>>>> I've attached the old and new repomd.xml files that I have. Both 
>>>> have the
>>>> same revision, but the timestamps of the data files are different.  
>>>> The order
>>>> and structure of the files are different as well so something 
>>>> reprocessed them.
>>>>
>>>> The latest timestamp in the old file is 1460563169 from 
>>>> updateinfo.  The
>>>> latest timestamp in the new file is 1460563168 from primary_db.  
>>>> updateinto is
>>>> not in the old repomd.xml file.
>>>>
>>>>
>>> Hmmmmm.....
>>>
>>> We use modifyrepo to inject the updateinfo data based on the packages
>>> within the metadata.
>>>
>>> I'll see about adjusting the process there.
>>
>> Is there any reason modifyrepo is used instead of just createrepo? I 
>> understand it may take longer to do a full (re)create, but I've never 
>> run into this problem when I've done repos.
>>
>> My workflow is rather simple, but I do:
>>
>> #!/bin/bash
>> if [ -z $1 ]; then
>>     echo "Usage: $0 <reponame>"
>>     exit 1
>> fi
>>
>> createrepo --outputdir=/path/to/repo/$1/x86_64/ /path/to/repo/$1/x86_64/
>>
>> rsync --delete-after --progress -a /path/to/$1/ 
>> livesite:/path/to/repo/$1/
>> exit 0
>>
>> You could optionally use the --update parameter to update an existing 
>> repo - which may be faster.
>>
>
> The createrepo command does not insert the updateinfo metadata even 
> when present within the repodata directory.  Like the group 
> information, it comes from an external data source, but unlike group 
> info there isn't a switch to load it in.
>
> I can look into switching to createrepo_c which has the relevant 
> switches, but that will take a bit as we'll need to modify the 
> publication workflow.
>
> I'd like to try one more possible workaround before investing the time 
> there.  Two createrepo commands feels a bit strange and like the kind 
> of thing I'd optimize out down the road if this isn't sufficiently 
> well documented.
>
> Pat

I've rebuild the 7 security repos with my workaround in place.

Pat