On 2016-04-15 02:45, Pat Riehecky wrote:
> On 04/14/2016 11:32 AM, Orion Poplawski wrote:
>> On 04/14/2016 08:54 AM, Pat Riehecky wrote:
>>> On 04/14/2016 09:35 AM, Orion Poplawski wrote:
>>>> On 03/30/2016 10:44 AM, Pat Riehecky wrote:
>>>>> On 03/30/2016 11:39 AM, Orion Poplawski wrote:
>>>>>> On 03/29/2016 08:55 AM, Orion Poplawski wrote:
>>>>>>> I get messages like this fairly frequently on our SL7 servers:
>>>>>>> 
>>>>>>> /etc/cron.daily/0yum-daily.cron:
>>>>>>> 
>>>>>>> Not using downloaded repomd.xml because it is older than what we 
>>>>>>> have:
>>>>>>>      Current   : Fri Mar 25 08:54:39 2016
>>>>>>>      Downloaded: Fri Mar 25 08:54:20 2016
>>>>>>> 
>>>>>>> Today, this is from the sl-security repo:
>>>>>>> 
>>>>>>> -rw-r--r--. 1 root root 3059 Mar 25 08:54
>>>>>>> /var/cache/yum/x86_64/7/sl-security/repomd.xml
>>>>>>> 
>>>>>>> We use a mirroring scheme that is based on time-stamps which may 
>>>>>>> be
>>>>>>> aggravating this locally, and are mirroring from
>>>>>>> http://ftp1.scientificlinux.org/linux/scientific
>>>>>>> 
>>>>>>> However it seems that the time-stamps of these files should never 
>>>>>>> go
>>>>>>> backwards.  Is there something going on in the SL staging process 
>>>>>>> causing
>>>>>>> this?
>>>>>> A little more information.  Apparently the <revision> tag in the
>>>>>> repomd.xml is
>>>>>> treated as the timestamp of the file.  Currently for the currently 
>>>>>> downloaded
>>>>>> sl-security the revision is:
>>>>>> 
>>>>>> $ grep revision /var/cache/yum/x86_64/7/sl-security/repomd.xml
>>>>>>      <revision>1458917662</revision>
>>>>>> $ ls -l /var/cache/yum/x86_64/7/sl-security/repomd.xml
>>>>>> -rw-r--r--. 1 root root 3059 Mar 25 08:54
>>>>>> /var/cache/yum/x86_64/7/sl-security/repomd.xml
>>>>>> $ date --date=@1458917662
>>>>>> Fri Mar 25 08:54:22 MDT 2016
>>>>>> $ stat --printf '%y\n' 
>>>>>> /var/cache/yum/x86_64/7/sl-security/repomd.xml
>>>>>> 2016-03-25 08:54:40.000000000 -0600
>>>>>> 
>>>>>> Now I don't understand how the cached copy ended up with a 
>>>>>> slightly newer
>>>>>> timestamp - possibly due to some munging by yum after download.
>>>>>> 
>>>>>> For the current version on my mirror:
>>>>>> 
>>>>>> # ls -l 7x/x86_64/updates/security/repodata/repomd.xml
>>>>>> -rw-r--r--. 1 apache apache 2965 Mar 29 07:46
>>>>>> 7x/x86_64/updates/security/repodata/repomd.xml
>>>>>> # grep revision 7x/x86_64/updates/security/repodata/repomd.xml
>>>>>>     <revision>1458917662</revision>
>>>>>> 
>>>>>> So despite being newer and different file, it has the same 
>>>>>> "revision" as the
>>>>>> older one.
>>>>>> 
>>>>>> Perhaps you are manually setting --revision in createrepo?  By 
>>>>>> default it
>>>>>> sets
>>>>>> revision to the unix timestamp at the start of the run.
>>>>>> 
>>>>> I'll alter the publication tools to add this value
>>>>> 
>>>>> Pat
>>>> Seems to have happened again:
>>>> 
>>>> # /etc/cron.daily/0yum-daily.cron
>>>> Not using downloaded repomd.xml because it is older than what we 
>>>> have:
>>>>     Current   : Wed Apr 13 09:59:29 2016
>>>>     Downloaded: Wed Apr 13 09:59:28 2016
>>>> 
>>>> -rw-r--r--. 1 root root 3059 Apr 13 09:59
>>>> /var/cache/yum/x86_64/7/sl-security/repomd.xml
>>>> 
>>> http://ftp.scientificlinux.org/linux/scientific/7x/x86_64/updates/security/repodata/repomd.xml
>>> 
>>> 
>>> Contains the revision field you requested.
>>> 
>>> Data from the publication filesystem:
>>> $ pwd
>>> /linux/scientific/7x/x86_64/updates/security/repodata
>>> $ stat repomd.xml
>>>    File: ‘repomd.xml’
>>>    Size: 2965          Blocks: 64         IO Block: 32768  regular 
>>> file
>>> Device: 2ah/42d    Inode: 905695246   Links: 1
>>> Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/ root)
>>> Context: system_u:object_r:nfs_t:s0
>>> Access: 2016-04-14 07:01:06.288136000 -0500
>>> Modify: 2016-04-13 11:06:36.036051000 -0500
>>> Change: 2016-04-13 11:06:36.036051000 -0500
>>>   Birth: -
>>> $
>>> 
>>> Pat
>> Okay, it appears that I misread some of the code originally.  It 
>> appears that
>> the "timestamp" of the repomd file is the latest timestamp of all of 
>> the
>> component data entries.
>> 
>> I've attached the old and new repomd.xml files that I have.  Both have 
>> the
>> same revision, but the timestamps of the data files are different.  
>> The order
>> and structure of the files are different as well so something 
>> reprocessed them.
>> 
>> The latest timestamp in the old file is 1460563169 from updateinfo.  
>> The
>> latest timestamp in the new file is 1460563168 from primary_db.  
>> updateinto is
>> not in the old repomd.xml file.
>> 
>> 
> Hmmmmm.....
> 
> We use modifyrepo to inject the updateinfo data based on the packages
> within the metadata.
> 
> I'll see about adjusting the process there.

Is there any reason modifyrepo is used instead of just createrepo? I 
understand it may take longer to do a full (re)create, but I've never 
run into this problem when I've done repos.

My workflow is rather simple, but I do:

#!/bin/bash
if [ -z $1 ]; then
     echo "Usage: $0 <reponame>"
     exit 1
fi

createrepo --outputdir=/path/to/repo/$1/x86_64/ /path/to/repo/$1/x86_64/

rsync --delete-after --progress -a /path/to/$1/ 
livesite:/path/to/repo/$1/
exit 0

You could optionally use the --update parameter to update an existing 
repo - which may be faster.

-- 
Steven Haigh

Email: [log in to unmask]
Web: https://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897