On 07/18/2012 01:58 PM, James Holland wrote:
>> Would love to see this testing process/code documented somewhere and
>> what the implications are. So we might be able to better and improve the
>> process in CentOS as well.
> 
> Is there a place where you document the CentOS processes?
> 

we are starting to, the QA team has been in place for about 4 years now
and we try to cover as many roles/deployment types as can. There is
quite a bit of documentation on the wiki, additions and contributions to
that are welcome.

I've personally spoken at Fosdem earlier this year on the Continuous
Integration process we are working on ( http://ci.dev.centos.org/ is the
'public dev' side for this, with the t_functional suite triggers in
place now ). The Distro/tree tests, which themselves number 150+, is
something I'm working on to get into the same harness before we hit
CentOS-5.9 / 6.4. Problem with that is as the other test suites before,
its been built in place to cater to a very specific setup working in a
very specific state - and because they consider the entire 'distro tree'
( ie. the mirror.centos.org layout ), its not exactly portable.  I hope
to have that fixed and onto the gitorious setup soon.

the actual test harness around it does dozens of installs trying to
cover every possible install role and type ( eg. we test xen, kvm,
virtualbox, vmware, real machines for http installs, iso installs, ftp
installs, nfs installs. And were using local as well as remote storage
in each of those setups ). If you now write up a matrix, just the
installer is getting well exercised. We're hoping to expand this into
the next release by adding more install environments and roles per
deployment (eg. most of the testing now is done for a base setup and the
test suites being run inside each base setup per environment ). The plan
is to expand that into addressing different roles ( please contribute
kickstarts! ) as well.

Its worth noting here that this is not something we do on a per-release
basis, the entire test suite is something we aim to run on every
individual package build + release.

For release time work, the QA team guys do manual tests over and above
everything else. To some extent, I feel the 'QA team' term is no longer
really relevant, these guys have mostly transitioned into being the
Release Team now.

As before, contributions and additions to the test harness, the test
suites, the automation and even ideas for testing etc, are welcome.

-- 
Karanbir Singh
+44-207-0999389 | http://www.karan.org/ | twitter.com/kbsingh
ICQ: 2522219    | Yahoo IM: z00dax      | Gtalk: z00dax
GnuPG Key : http://www.karan.org/publickey.asc