Ken Teh said...

|NICs are activated in order eth0, eth1, ...  What gets called eth0, eth1, 
|... is determined by the order in which they are "discovered" which is 
|usually just the PCI order on the motherboard.

But therein lies the problem.  The
discovery order is at least somewhat
random (or close enough that it's
irrelevant whether it truly is).

We have several servers with two
to four NICs.  Let's say that on
first boot of a 4 NIC system, the
NICs come up, bottom to top, as 0,
1, 2, 3.  After a prolonged power
failure that drains the UPS, they
come up 0, 2, 1, 3.  We figure this
out and move the cables.  A couple
of months later, the system locks up
when the PS dies.  New PS, and they
come up 3, 2, 1, 0.  We discover this
and move the cables.  A few months
later, we have to add more RAM and
disk.  This time they come up 0, 1,
2, 3 again.

This is one real life example.  The
numbers aren't necessarily exactly
what we saw, but they're representative.
Another four NIC system rebooted
fine 2 or 3 times with the same order
before showing similar behavior.

I don't believe this was the case
with RedHat 7.1, but I wouldn't swear
to it.

I have bought a 4 port card from
Soekris to see if having all the
ports on one card solves the problem,
but haven't gotten to test it yet.