In-line...
> -----Original Message-----
> From: [log in to unmask]
> [mailto:[log in to unmask]] On
> Behalf Of [log in to unmask]
> Sent: 27 March 2008 16:07
> To: [log in to unmask]
> Subject: Re: fsck.ext3 on large file systems?
>
> On Thu, 27 Mar 2008, Stephen John Smoogen wrote:
>
> > On Wed, Mar 26, 2008 at 7:34 PM, Michael Hannon
> <[log in to unmask]> wrote:
> >> Greetings. We have a lately had a lot of trouble with
> relatively large
> >> (order of 1TB) file systems mounted on RAID 5 or RAID 6
> volumes. The
> >> file systems in question are based on ext3.
> >>
> >> In a typical scenario, we have a drive go bad in a RAID
> array. We then
> >> remove it from the array, if it isn't already, add a new
> hard drive
> >> (i.e., by hand, not from a hot spare), and add it back to the RAID
> >> array. The RAID operations are all done using mdadm.
This is of course a software RAID...
> >> After the RAID array has completed its rebuild, we run
> fsck on the RAID
> >> device. When we do that, fsck seems to run forever,
> i.e., for days at a
> >> time, occasionally spitting out messages about files with
> recognizable
> >> names, but never completing satisfactorily.
> >>
> >
> > fsck of 1TB is going to take days due to the linear nature of it
>
> Hmm, we successfully fsck'd ext3 filesystems 1.4 TB in size
> frequently a
> couple of years ago, under 2.4 (back then, it was SuSE 8.2 +
> a vanilla
> kernel). This took no more than a few hours (maybe 2,3, or
> 4). It was
> hardware RAID, not too reliable (hence "frequently"), and not
> too fast (<
> 100 MB/s). A contemporary linux server with software RAID
> should complete
> an fsck *much* faster, or something is wrong.
On our hardware RAID arrays (3ware, Areca, Infortrend) with many (12/14)
SATA disks, 500/750GB each, we fsck 2TB+ ext3 filesystems (as
infrequently as possible!) and it takes ~2 hours each. We have some
5.5TB arrays that take less than three hours. Note that these are
created with '-T largefile4 -O dir_index' among other options.
I'd be very suspicious of a HW RAID controller that took 'days' to fsck
a file system unless the filesystem was already in serious trouble, and
from bitter experience, fsck on a filesystem with holes it it caused by
a bad raid controller interconnect (SCSI!) can do more damage than good.
Maybe the controller has some missconfiguration.
Not so sure about software raid setups though. But if you have a lot of
disks on a single controller chip, say driving 4 SATA disks, you *might*
max out the PCI(-e/X) bandwidth to the controller chip.
One thing that will kill you though - if you have a lot of remapped
blocks on your device, that can slow things down considerably. The
device may be OK but performance would suck. Another thing that some
harwdare RAID controllers do is turn off the device cache in some
circumstances - that bit us a while back and made a controller crawl
(BBU failed, controller went into 'safe' mode).
> And I still wonder why fsck at at all just because a broken disk was
> replaced in a redundant array?
Yes - should not be necessary, but I've done it for super-critical
(small) systems 'just in case' due to paranoid user groups.
> --
> Stephan Wiesand
> DESY - DV -
> Platanenallee 6
> 15738 Zeuthen, Germany
Martin.
--
Martin Bly
RAL Tier1 Fabric Team
|
