On Wed, Mar 26, 2008 at 7:34 PM, Michael Hannon <[log in to unmask]> wrote:
> Greetings.  We have a lately had a lot of trouble with relatively large
>  (order of 1TB) file systems mounted on RAID 5 or RAID 6 volumes.  The
>  file systems in question are based on ext3.
>
>  In a typical scenario, we have a drive go bad in a RAID array.  We then
>  remove it from the array, if it isn't already, add a new hard drive
>  (i.e., by hand, not from a hot spare), and add it back to the RAID
>  array.  The RAID operations are all done using mdadm.
>
>  After the RAID array has completed its rebuild, we run fsck on the RAID
>  device.  When we do that, fsck seems to run forever, i.e., for days at a
>  time, occasionally spitting out messages about files with recognizable
>  names, but never completing satisfactorily.
>

fsck of 1TB is going to take days  due to the linear nature of it
checking the disk. [ I think the disks for mirrors.kernel.org take
many weeks to fsck.] The bigger question is what kind of data are you
writing to these disks, and is the ext3 journal large enough for those
writes?


>  The systems in question are typically running SL 4.x.  We've read that
>  the version of fsck that is standard in SL 4 has some known bugs,
>  especially wrt large file systems.
>
>  Hence, we've attempted to repeat the exercise with fsck.ext3 taken from
>  the Fedora 8 distribution.  This gives us improved, but still not
>  satisfactory, results.
>

Did you recompile the binary from source, or did you use it straight?
I am just wondering if fsck is dependant on some kernel particulars...

To tell you the truth, I have not done anything with Linux Raid in the
Terabyte range.. Usually I go with a hardware solution at that point
(usually for business reasons.. that much storage usually comes with a
box with hardware raid). I did run into a similar issue though trying
to help someone last week on a SuSE box with ext3. They also had a
long fsck and weird file names coming up. I think they went with the
same solution ( restore from backups).



-- 
Stephen J Smoogen. -- CSIRT/Linux System Administrator
How far that little candle throws his beams! So shines a good deed
in a naughty world. = Shakespeare. "The Merchant of Venice"