Hi,
On Mon, 14 Aug 2006, Pann McCuaig wrote:
> Greetings!
>
> I'm having an occasional problem with one host in a cluster of nine. The
> eight hosts that are not giving me this headache are identical (except
> for amount of RAM) Sun Fire V20z with dual Opteron 850 CPUs and either
> 4G or 8G of RAM. The problem host is our "big iron."
Those 8 V20z, do they have a megaraid RAID-5 as well?
> Platform Information
> --------------------
> Host: Sun Fire V40z Server with 4 * Opteron 850 CPU and 32GB RAM
>
> OS: Scientific Linux 4.3, 2.6.9-34.0.1.ELsmp x86_64 kernel
>
> Controller: Sun MegaRAID 320-2X Dual Ultra-320 SCSI Card (p/n X9269A)
>
> Drives: 5 * 146GB 10K RPM Ultra320 SCSI Hard Drive (p/n X9257A)
>
> Host: scsi1 Channel: 01 Id: 06 Lun: 00
> Vendor: SDR Model: GEM318P Rev: 1
> Type: Processor ANSI SCSI revision: 02
>
> Host: scsi1 Channel: 02 Id: 00 Lun: 00
> Vendor: MegaRAID Model: LD 0 RAID5 560G Rev: 413G
> Type: Direct-Access ANSI SCSI revision: 02
>
> Configuration is RAID 5 and I took all the defaults when setting it up.
>
> /dev/sda1 on / type ext3 (rw)
> /dev/sda4 on /tmp type ext3 (rw)
> /dev/sda2 on /var type ext3 (rw)
>
> Filesystem Size Used Avail Use% Mounted on
> /dev/sda1 7.9G 2.0G 5.5G 27% /
> /dev/sda4 437G 18G 398G 5% /tmp
> /dev/sda2 63G 129M 60G 1% /var
>
> The nature of the problem is an apparent "hang" for some small finite
> period of time (somewhere between ten minutes and two hours according
> to user accounts). When the host is "hung," response is very slow. Load
> average is quite high (above 10), and % wa is high (~40%). kjournald
> is always near the "top" of top, but doesn't appear to be using many
> resources, either %CPU or %MEM. But it's always there when the host is
> "hung," and when things are running "normally," it only puts in the
> occasional appearance.
>
> I really don't know how often it happens. One user reported it happening
> on two consecutive days, but this host is used primarily for big SAS or
> Stata or Matlab jobs that run in the background (sometimes for days) so
> that the "hangs" could happen fairly frequently without being noticed.
> That very large /tmp partition is for the use of these programs; /home
> is on NFS (gigabit ethernet) and our users have learned they get much
> better response using the local drive.
>
> The "hang" always "repairs itself" without human intervention.
I'm seeing problems that are at least ver very similar, with a different
MegaRAID Model. They started when the controllers came with a new firmware
release. Older ones do not have the problem (we own 20 such systems).
I still haven't pinned it down completely, but I think the "hangs"
(nothing actually hangs, I/O just becomes very slow - hdparm -tT shows
< 1 MB/s instead of ~ 200) are due to a firmware feature called "Patrol
Reads".
Patrol Reads continuously scan the disks in the background. This is good,
but sometimes the controller seems to be getting the priorities wrong.
Upgrading to the very latest firmware makes the problem happen less
frequently (and hence even harder to detect and understand), but doesn't
solve it completely. Turning off Patrol Reads during a "hang" immediately
brings things back to normal.
Check whether your firmware has this feature, and whether it's active.
You may have to do it in the controller BIOS unless SUN provides up to
date management software for the adapter that works on linux, or you find
something recent matching your card on the lsil support site.
Also, are you running MegaCtrl?
My current plan for my systems is to turn these background scrubs off
and return to weekly consistency checks only.
NB since you said you're new to RAID: Making sure that every block on each
disk in the array is read once in a while (weekly scans are sufficient
from my experience) is a very good idea. Hence such features shouldn't
be turned off blindly. It's just that consistency checks give me a very
minor slowdown only, while Patrol Reads seem to have a problem. LSI
recommends using both, and that's what I would prefer if possible.
> My working theory is that there is some sort of negative interaction
> between kjournald and the RAID driver; disk accesses (or at least
> writes) are being inhibited while kjournald goes about its business.
I guess it's rather an interaction between the driver and the firmware.
> I'm looking for any suggestions about how to troubleshoot the problem.
Hope this helps. Please let me know what you find out.
Cheers,
Stephan
> I'm reasonably knowledgeable about Linux, but this is my first
> experience with RAID. Pointers to any FMs I should R are welcome.
>
> Cheers,
> Pann
>
--
Stephan Wiesand
DESY - DV - Phone: +49 33762 7 7370
Platanenallee 6 Fax: +49 33762 7 7216
15738 Zeuthen, Germany
|
