On Sun, Nov 29, 2009 at 8:07 AM, Stijn De Weirdt
<[log in to unmask]> wrote:
> hi all,
>
>
> from my limited knowledge, as far as kernel is repsonsible for the
> difference (between RHEL and CentOS or SL), it seems stock sl5.4 kernel
> is build with a few settings that are not so in favour for hpc (like
> desktop preemption and 1000Hz CPU_FREQ) and some others that could make
> a difference (build for size (-Os instead of -O2) and with debuginfo
> build in).
Yes good points. There are tunables that can be set for HPC
environments. In my tests I was utilizing the default kernel as
provided by upstream.
>
> if someone has some single machine benchmarks that could be used to
> verify the influence, i can spend some time in trying out.
I would recommend using MPI or OpenSM capable benchmarks and using all
of the cores on the system.
Here are a few potential starting points:
http://www.cs.virginia.edu/stream/
http://icl.cs.utk.edu/projects/llcbench
http://www.nas.nasa.gov/Resources/Software/npb.html
And lastly, I love the traditional "time make -j
[1,2,4,8,16,32,64,128,256]" in a kernel source tree (with a "make
mrproper" inbetween and the same .config).
>
>> My tests were done implementing Perceus for cluster management, and I
>
> which applications did you use for testing?
In particular I tested with an Oil/Gas typical workload that was very
tightly coupled with lots of MPI barriers and interprocess
communication (it isn't redistributable).
HPL is another one that I tested, but it isn't tightly coupled so the
deltas were not as significant.
>
>> The reason for the difference that we found has to do primarily with
>> the build environment that Red Hat uses. RHEL is not built on RHEL but
>> rather a modified and highly optimized tool chain that is not
>> distributed. It would be reasonable to assume that Fedora is also
>> built using this environment.
>>
> which parts of the OS should be affected by this the most?
In my tests, the C library was the main bottleneck but there is no
reason to think that this doesn't affect the entire system to a lesser
degree.
Also, as I mentioned, for serial apps and standalone runs I didn't see
a very noticeable difference (but that wasn't the focus of my tests).
>
>> BTW, this is one of the reasons that after founding Centos I continued
>> to also focus on a freely available high performance focused
>> distribution of Linux (Caos Linux) that is tuned and tested for high
>> performance (but lacks many of the packages as it is not general
>> purpose, which for those requirements I now run SL).
>>
> nice, i'll definitely have a look at it.
For HPC and servers it works well, but keep in mind it isn't intended
as a general purpose Linux solution. Let me know what you think. :)
Greg
--
Greg M. Kurtzer
Chief Technology Officer
HPC Systems Architect
Infiscale, Inc. - http://www.infiscale.com
|
