Note that my prior comments about ideal EL6 build process (quoted below) 
no doubt contain many error beyond the couple I'll self-reply to quickly 
here.  The main point I was making was trying to shed appropriate light 
on the actual scale, and non-obvious-simplicity of the problem.

First, where I alluded to 'choice b' I meant 'decision b' as a couple 
sentences later I define an actual choice b.  Second, the bit about 
early dogfooding/rebootstrapping the minbuildroot is actually not really 
all that useful, as the current SL6 (and what I equivalently encoded in 
my el-build script), actually uses *FOUR* bootstrap OSs for the first 
pointrelease(6.0) build.  I.e. using F12/F13/RHEL6Beta/SL to be able to 
build all of the 6.0 packages.  So rebootstrapping the minbuildroot as I 
described, while sounding good in some ideal theory way, probably won't 
work without significant alternate workarounds to the issues that 
currently require F12/F13/RHEL6Beta for various package build deps. 
Thus where I would want to put in that first dogfooding/rebootstrapping 
phase would be after getting a complete 6.0 package set, rebuilding 
another complete 6.0 package set against only that first output.  I 
haven't yet tried this, but it seems pretty clearly desirable even if 
one has to modify the package build scripts (SPECS) to get that to work. 
  Though if you're willing to modify specs for that purpose, then it 
becomes more likely you'd modify them such that you don't need all those 
F12/F13/RHEL6beta initial bootstrap binaries (thus reducing your input 
complexity/threat surface).

Anyway...  again, just trying to give people a feel for the real scope 
of the issues at hand.  My own engineering strategy is to get _any_ 
deterministic process laid down and producing remotely-reasonable output 
first, then to easily go and tweak the fully automated/coded build 
process towards a more optimal/puritanical path iteratively after that. 
  Ideally such that by the time EL7 comes out, the turnaround time for 
updating the deterministic buildscripts is really short, because all 
these issues have been tackled and solved in code for EL6 already.

-dmc


On 01/20/2012 08:41 PM, Douglas McClendon wrote:
> On 01/20/2012 07:43 AM, Michael Tiernan wrote:
>> On 1/19/12 6:17 PM, Douglas McClendon wrote:
>>> It's noteworthy that I spent several months recently trying to more or
>>> less recreate the SL6 build in a completely automated way for
>>> Ascendos(.org) after Troy Dawson got the ball rolling on that project.
>>> At the moment I've sort of thrown up my hands for awhile, as there
>>> didn't seem to be sufficient interest in my particular brand of
>>> solution and/or leadership.
>>
>> Oh boy. That doesn't sound good.
>>
>> I appreciated the work that was started on ascendos but I for one knew
>> little of the overall picture and was/am unable to read hundreds of
>> forum posts about what's going on. It's just that 24hrs/day problem.
>>
>> Maybe this discussion will help trigger a renewed push to the community.
>>
>>> And in general, is ScientificLinux capable of describing a
>>> deterministic algorithm used to make those choices,
>>
>> I guess that's the summary of the problem right there. My friend has
>> suggested that he's going to build a virtual box system that contains a
>> virtual machine that he can pull out at any given moment, that's
>> 'virgin' and say "build from scratch" and get an output that, other than
>> the dates involved is identical to the existing distribution.
>
> That is essentially how I designed and implemented 'el-build', except
> using kvm instead of virtualbox, as kvm/virsh is what is more of a
> stock(/RH-encouraged) solution found on e.g. an SL5 or F13 system.
>
> Thus the idea (preaching to the choir here obviously) being that if you
> are on any libvirt/kvm capable system (e.g. centos5/debuntu/f13) you can
> easily do the pushbutton fully automated and deterministic build. And
> that within the VM, the building environment OS (call it the MockHostOS)
> is a known/determin{ed,istic} quantity. I actually chose to use SL60 as
> MockHostOS because I that is what Troy started Ascendos's initial koji
> with, and it happened to exist, and seemed likely to make things go
> easiest. But the long term plan was once that was completely working,
> able to build all of EL6->currentupdates, I would then make the few
> tweaks necessary to use F13 instead as the MockHostOS, as my best guess
> was that was what SL used for SL6. (actually maybe you folks used SL5,
> but even were that the case I'd still want to go with F13 because it
> feels like a less total bootstrap complexity to me, and in line with
> what I expected RH used. Or rather maybe RH used EL5, in which case then
> I'm tilting between CentOS5 to match that but with added complexity
> surface*, vs F13).
>
> (surface of code and builds making up the initial bootstrap binaries,
> i.e. whether or not security breaches within the centos/SL environments
> factor into the security of the bootstrap binaries. Yes, I'm that paranoid)
>
> Anyway, thats a bunch of rambling, but its also I think the root of the
> important 'deterministic genesis' conversation which I think SL should
> have dedicated more time and attention to by now. (note my shameless
> naming of my own engineering price in the grandios email to ascendos-dev
> that I got lwn to post/reprint/reporton, with anti-Gitmo politicization
> intact).
>
> Anyway, so el-build currently uses SL60 as the MockHostOS it first
> installs in its virtual build system.
>
> Then within that, it's currently using the standard koji/mock toolset,
> entirely facilitated by the koji setup scripts from ScientificLinux
> (thanks, though after many months and decent introduction and
> understanding of koji, I'm not entirely sure its the right tool for this
> job. Though no doubt entirely sufficient.).
>
> It gets interesting of course then, as koji/mock is setting up the
> buildroots.
>
> The first interesting question there (I think) which I've deferred thus
> far in code (I think, been a few weeks since I've been looking at the
> code), is whether or not you do an immediate dogfooding pass on the min
> buildroot. I.e.
>
> choice a) your mock/minbuildroot deps are satisified by your core
> bootstrap OS (call it FirstMinBuildRootOS). Then further deps are
> satisified by your 6.0 output packages as they get built (* in what
> order? call that choice b, I went with reverse datesorted by srpm build
> date. pkg by pkg, not weekly ala SL which I don't think is preferrable)
>
> choice b) you just build the set of packages in that
> FirstMinBuildRootOS, then immediately cease using that first bootstrap
> minbuildroot and start exclusively using output, call that now
> SecondMinBuildRootOS. You could even throw in another dogfooding pass
> (ThirdMinBuildRootOS) just for
> fun/pickiness/goodquickenoughsanitytest/etc..)
>
> Clearly by my wording I like choice b, though as mentioned I may have
> deferred it.
>
> Anyway, then the more fun choices comes when you start getting to past
> the first point release, and into updates, and again, certain options
> seem intuitively obvious and I either went with those, or took some
> easier route with a TODO in the code noting the long-term preferred
> solution. (i.e. there is an option to switch your virt host to the new
> pointrelease, or not. Though here the fact that things are done
> virt-wise allows that to be possible and have a single end-to-end build
> that doesn't require physically rebooting any system)
>
> Another key aspect of my solution design was the complete seperation of
> the first online-source-and-bootstrap-materials fetching phase of
> things, and the second actual building phase of things, such that the
> entire build process can be done on a pure virgin system never needing
> to be defiled by connecting it to the internets.
>
> Yet another core idea I've had and been quite unflinching about is the
> idea that the private part of the package signing key should never be
> accessible to a system that has ever or ever will be online. I kindof
> doubt SL (or even centos or redhat) do this.
>
> But all of this is a slice of a clearly larger discussion, that I'm a
> tad offended SL has seemed to ignore up until now (given my clear
> efforts, and even the Troy organizational connection). (But yes, I know,
> no budget, expedience, people have lives outside work they want to live
> and not worry about nitpicking esoteric techno stuff like this, etc...)
>
> But since you gave me an opening to step back up on my soapbox... :)
>
> cheers,
>
> -dmc
>
>
>
>>
>>> Anyway, for people interested in the landscape of this issue, this
>>> recent lwn article (and my arguably inflammatory attached comments)
>>> has some perspective on the issue.
>>
>> I'll read this later today, no way I can get it done soon.