On Mon, 2004-10-25 at 15:32, Ken Teh wrote:
> Hello all,
>
> I'm looking to buy some hardware to set up a little farm for doing data
> analysis and numerical computation.  We've been conservative and used mostly
> Pentium-4 desktop workstations.  But, I'm curious about the SMP Opterons.  I
> hear mainly good things about them in the press.  I'd like to hear what you
> have to say.  Recommendations, gotchas, that sort of thing...  Also, your
> favorite vendors.

I have installed, and maintain, several Opteron clusters.

One of the first in the UK was installed last year,
a 46 node (92 CPU) cluster for computational chemistry.
Arima HDAMA motherboards.

Recently have installed a 135 node (270 CPU) plus
two high-availability master nodes cluster at Durham University.
This has 32 nodes with Myrinet networking.

Also recently a 22-way dual Opteron with Myrinet.

We also have others.



I guess I would make the following observations:

a) go for Opteron. They are stable, and run your existing 32 bit codes
out of the box. Our customers just ran their existing chemistry codes
right off.

(You can even just install a 32 bit Linux distro and run that - we have
done that at an early stage when the x86-64 distros weren't all that
stable last year. But a 'proper' 64 bit Linux is OK now.)

b) Many vendors will provide 1U dual Opterons.
   The popular motherboards are Tyan S2882 and Newisys.
   We provide Tyan.
   There's been a recent discussion on the Beowulf list on this choice.
   The Newisys board is slightly larger, so I gather you have to get a
   case which will take it.

c) You can get 4-way Opterons, we have them in 2U, and I gather there
   are 1U options out there.

d) Watch the cooling with Opterons - make sure your vendor is proposing
   a case with good cooling.


e) You might also wish to consider the Emt64/Nocoma systems from Intel
which are now coming out.
Nocoma clock at 3.4Ghz versus 2.4Ghz, but of course makign comparisons
like that is stupid. Nocoma doesn't have hypertransport.
I guess like every one will say - benchmark your own codes on each.
Opterons do very well where lots of memory accesses are taking place.

f) Which compilers do you use?

g) Go for rackmount cases if at all possible, with 400Watt PSUs.
   Desktops tend to have weaker PSUs, and also the cabling can rapidly
become a nightmare.


If I were you, I'd contact vendors in the US and spec a rackmount
system, with dual Opteron 250s (or 246 depending on budget) in a 1U
case.
Tyan motherboards, or Newisys.
The HDAMA ones we have in service have been reliable for over a year.
Its just that I've never got lm_sensors working reliably on them!