On Sun, Nov 29, 2009 at 8:07 AM, Stijn De Weirdt <[log in to unmask]> wrote: > hi all, > > > from my limited knowledge, as far as kernel is repsonsible for the > difference (between RHEL and CentOS or SL), it seems stock sl5.4 kernel > is build with a few settings that are not so in favour for hpc (like > desktop preemption and 1000Hz CPU_FREQ) and some others that could make > a difference (build for size (-Os instead of -O2) and with debuginfo > build in). Yes good points. There are tunables that can be set for HPC environments. In my tests I was utilizing the default kernel as provided by upstream. > > if someone has some single machine benchmarks that could be used to > verify the influence, i can spend some time in trying out. I would recommend using MPI or OpenSM capable benchmarks and using all of the cores on the system. Here are a few potential starting points: http://www.cs.virginia.edu/stream/ http://icl.cs.utk.edu/projects/llcbench http://www.nas.nasa.gov/Resources/Software/npb.html And lastly, I love the traditional "time make -j [1,2,4,8,16,32,64,128,256]" in a kernel source tree (with a "make mrproper" inbetween and the same .config). > >> My tests were done implementing Perceus for cluster management, and I > > which applications did you use for testing? In particular I tested with an Oil/Gas typical workload that was very tightly coupled with lots of MPI barriers and interprocess communication (it isn't redistributable). HPL is another one that I tested, but it isn't tightly coupled so the deltas were not as significant. > >> The reason for the difference that we found has to do primarily with >> the build environment that Red Hat uses. RHEL is not built on RHEL but >> rather a modified and highly optimized tool chain that is not >> distributed. It would be reasonable to assume that Fedora is also >> built using this environment. >> > which parts of the OS should be affected by this the most? In my tests, the C library was the main bottleneck but there is no reason to think that this doesn't affect the entire system to a lesser degree. Also, as I mentioned, for serial apps and standalone runs I didn't see a very noticeable difference (but that wasn't the focus of my tests). > >> BTW, this is one of the reasons that after founding Centos I continued >> to also focus on a freely available high performance focused >> distribution of Linux (Caos Linux) that is tuned and tested for high >> performance (but lacks many of the packages as it is not general >> purpose, which for those requirements I now run SL). >> > nice, i'll definitely have a look at it. For HPC and servers it works well, but keep in mind it isn't intended as a general purpose Linux solution. Let me know what you think. :) Greg -- Greg M. Kurtzer Chief Technology Officer HPC Systems Architect Infiscale, Inc. - http://www.infiscale.com