On 04/14/2016 08:20 PM, Steven Haigh wrote:
> On 2016-04-15 02:45, Pat Riehecky wrote:
>> On 04/14/2016 11:32 AM, Orion Poplawski wrote:
>>> On 04/14/2016 08:54 AM, Pat Riehecky wrote:
>>>> On 04/14/2016 09:35 AM, Orion Poplawski wrote:
>>>>> On 03/30/2016 10:44 AM, Pat Riehecky wrote:
>>>>>> On 03/30/2016 11:39 AM, Orion Poplawski wrote:
>>>>>>> On 03/29/2016 08:55 AM, Orion Poplawski wrote:
>>>>>>>> I get messages like this fairly frequently on our SL7 servers:
>>>>>>>>
>>>>>>>> /etc/cron.daily/0yum-daily.cron:
>>>>>>>>
>>>>>>>> Not using downloaded repomd.xml because it is older than what 
>>>>>>>> we have:
>>>>>>>>      Current   : Fri Mar 25 08:54:39 2016
>>>>>>>>      Downloaded: Fri Mar 25 08:54:20 2016
>>>>>>>>
>>>>>>>> Today, this is from the sl-security repo:
>>>>>>>>
>>>>>>>> -rw-r--r--. 1 root root 3059 Mar 25 08:54
>>>>>>>> /var/cache/yum/x86_64/7/sl-security/repomd.xml
>>>>>>>>
>>>>>>>> We use a mirroring scheme that is based on time-stamps which 
>>>>>>>> may be
>>>>>>>> aggravating this locally, and are mirroring from
>>>>>>>> http://ftp1.scientificlinux.org/linux/scientific
>>>>>>>>
>>>>>>>> However it seems that the time-stamps of these files should 
>>>>>>>> never go
>>>>>>>> backwards.  Is there something going on in the SL staging 
>>>>>>>> process causing
>>>>>>>> this?
>>>>>>> A little more information.  Apparently the <revision> tag in the
>>>>>>> repomd.xml is
>>>>>>> treated as the timestamp of the file.  Currently for the 
>>>>>>> currently downloaded
>>>>>>> sl-security the revision is:
>>>>>>>
>>>>>>> $ grep revision /var/cache/yum/x86_64/7/sl-security/repomd.xml
>>>>>>>      <revision>1458917662</revision>
>>>>>>> $ ls -l /var/cache/yum/x86_64/7/sl-security/repomd.xml
>>>>>>> -rw-r--r--. 1 root root 3059 Mar 25 08:54
>>>>>>> /var/cache/yum/x86_64/7/sl-security/repomd.xml
>>>>>>> $ date --date=@1458917662
>>>>>>> Fri Mar 25 08:54:22 MDT 2016
>>>>>>> $ stat --printf '%y\n' 
>>>>>>> /var/cache/yum/x86_64/7/sl-security/repomd.xml
>>>>>>> 2016-03-25 08:54:40.000000000 -0600
>>>>>>>
>>>>>>> Now I don't understand how the cached copy ended up with a 
>>>>>>> slightly newer
>>>>>>> timestamp - possibly due to some munging by yum after download.
>>>>>>>
>>>>>>> For the current version on my mirror:
>>>>>>>
>>>>>>> # ls -l 7x/x86_64/updates/security/repodata/repomd.xml
>>>>>>> -rw-r--r--. 1 apache apache 2965 Mar 29 07:46
>>>>>>> 7x/x86_64/updates/security/repodata/repomd.xml
>>>>>>> # grep revision 7x/x86_64/updates/security/repodata/repomd.xml
>>>>>>>     <revision>1458917662</revision>
>>>>>>>
>>>>>>> So despite being newer and different file, it has the same 
>>>>>>> "revision" as the
>>>>>>> older one.
>>>>>>>
>>>>>>> Perhaps you are manually setting --revision in createrepo?  By 
>>>>>>> default it
>>>>>>> sets
>>>>>>> revision to the unix timestamp at the start of the run.
>>>>>>>
>>>>>> I'll alter the publication tools to add this value
>>>>>>
>>>>>> Pat
>>>>> Seems to have happened again:
>>>>>
>>>>> # /etc/cron.daily/0yum-daily.cron
>>>>> Not using downloaded repomd.xml because it is older than what we 
>>>>> have:
>>>>>     Current   : Wed Apr 13 09:59:29 2016
>>>>>     Downloaded: Wed Apr 13 09:59:28 2016
>>>>>
>>>>> -rw-r--r--. 1 root root 3059 Apr 13 09:59
>>>>> /var/cache/yum/x86_64/7/sl-security/repomd.xml
>>>>>
>>>> http://ftp.scientificlinux.org/linux/scientific/7x/x86_64/updates/security/repodata/repomd.xml 
>>>>
>>>>
>>>>
>>>> Contains the revision field you requested.
>>>>
>>>> Data from the publication filesystem:
>>>> $ pwd
>>>> /linux/scientific/7x/x86_64/updates/security/repodata
>>>> $ stat repomd.xml
>>>>    File: ‘repomd.xml’
>>>>    Size: 2965          Blocks: 64         IO Block: 32768 regular file
>>>> Device: 2ah/42d    Inode: 905695246   Links: 1
>>>> Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: ( 0/ root)
>>>> Context: system_u:object_r:nfs_t:s0
>>>> Access: 2016-04-14 07:01:06.288136000 -0500
>>>> Modify: 2016-04-13 11:06:36.036051000 -0500
>>>> Change: 2016-04-13 11:06:36.036051000 -0500
>>>>   Birth: -
>>>> $
>>>>
>>>> Pat
>>> Okay, it appears that I misread some of the code originally. It 
>>> appears that
>>> the "timestamp" of the repomd file is the latest timestamp of all of 
>>> the
>>> component data entries.
>>>
>>> I've attached the old and new repomd.xml files that I have. Both 
>>> have the
>>> same revision, but the timestamps of the data files are different.  
>>> The order
>>> and structure of the files are different as well so something 
>>> reprocessed them.
>>>
>>> The latest timestamp in the old file is 1460563169 from updateinfo.  
>>> The
>>> latest timestamp in the new file is 1460563168 from primary_db.  
>>> updateinto is
>>> not in the old repomd.xml file.
>>>
>>>
>> Hmmmmm.....
>>
>> We use modifyrepo to inject the updateinfo data based on the packages
>> within the metadata.
>>
>> I'll see about adjusting the process there.
>
> Is there any reason modifyrepo is used instead of just createrepo? I 
> understand it may take longer to do a full (re)create, but I've never 
> run into this problem when I've done repos.
>
> My workflow is rather simple, but I do:
>
> #!/bin/bash
> if [ -z $1 ]; then
>     echo "Usage: $0 <reponame>"
>     exit 1
> fi
>
> createrepo --outputdir=/path/to/repo/$1/x86_64/ /path/to/repo/$1/x86_64/
>
> rsync --delete-after --progress -a /path/to/$1/ 
> livesite:/path/to/repo/$1/
> exit 0
>
> You could optionally use the --update parameter to update an existing 
> repo - which may be faster.
>

The createrepo command does not insert the updateinfo metadata even when 
present within the repodata directory.  Like the group information, it 
comes from an external data source, but unlike group info there isn't a 
switch to load it in.

I can look into switching to createrepo_c which has the relevant 
switches, but that will take a bit as we'll need to modify the 
publication workflow.

I'd like to try one more possible workaround before investing the time 
there.  Two createrepo commands feels a bit strange and like the kind of 
thing I'd optimize out down the road if this isn't sufficiently well 
documented.

Pat