On Wed, 26 Mar 2008, Jon Peatfield wrote:

> On Wed, 26 Mar 2008, Jan Schulze wrote:
>
>> Hi all,
>> 
>> I have a disk array with about 4.5 TB and would like to use it as one large 
>> logical volume with an ext3 file system. When mounting the logical volume , 
>> I get an "Input/Output error: can't read superblock".
>
> Do you get any interesting kernel messages in the output of dmesg (or 
> /var/log/messages etc)?  Which exact kernel is this (uname -r) and what arch 
> (i386/x86_64 etc; uname -m)?

And what driver/hardware?

>> I'm using SL 4.2 with kernel 2.6 and this is what I did so far:
>> 
>> - used parted to create a gpt disk label (mklabel gpt) and one large
>>   partition (mkpart primary ext3 0s -1s)
>> 
>> - used parted to enable LVM flag on device (set 1 LVM on)
>
> I know it would be slow but can you test that you can read/write to all of 
> /dev/sda1?

Using dd's "seek" parameter, this should not take too much time. But if
creating the GPT label & partition was successful, chances are the whole 
device is accessible.

>> - created one physical volume, one volume group and one logical volume
>>   (pvcreate /dev/sda1, vgcreate raid6 /dev/sda1, lvcreate -l 1189706 -n
>>   vol1 raid6)
>> 
>> - created an ext3 filesystem and explicitly specified a 4K blocksize, as
>>   this should allow a filesystem size of up to 16 TB (mkfs.ext3 -m 0 -b
>>   4096 /dev/raid6/vol1)
>
> For some reason my EL4 notes tell me that we also specify -N (number of 
> inodes), as well as -E (set RAID stride), -J size= (set journal size) and -O 
> sparse_super,dir_index,filetype though most of that is probably the default 
> these days...

Specifying the stripe width is also supposed to be a good idea, as is 
aligning the start of the partition to a stripe boundary (although that's 
more likely to be useful without LVM on top).

>> However, mounting (mount /dev/raid6/vol1 /raid) gives the superblock error, 
>> mentioned above.
>> 
>> Everything is working as expected, when using ext2 filesystem (with LVM) or 
>> ext3 filesystem (without LVM). Using a smaller volume (< 2 TB) is working
>> 
>> with ext3+LVM as well. Only the combination of > 2TB+ext3+LVM gives me 
>> trouble.
>> 
>> Any ideas or suggestions?
>
> We found that in at least some combinations of kernel/hardware (drivers 
> really I expect), that support for >2TB block devices was still rather flakey 
> (at least in the early versions of EL4).
>
> We ended up getting our RAID boxes to present as multiple LUNs each under 2TB 
> which we can then set up as PVs and join back together into a single VG and 
> still have an LV which was bigger than 2TB.  I'm rather conservative in such 
> things so we still avoid big block devices at the moment.
>
> [ obviously with single disk sizes growing at the rate they are it means that 
> the block-devices >2TB code is going to get a LOT more testing! ]

We're successfully using devices up to 7 TB with a single XFS 
filesystem on them, under SL4/5 (but I think we started doing this with 
4.3, not 4.2). I have no hope to be able to check (xfs_repair) those 
should this ever become necessary though - from what I've read it would 
require more RAM than fits into a server today.

> However, some of the tools (e.g. ext2/3 fsck) still seemed to fail at about 
> 3.5TB so we ended up needing to build the 'very latest' tools to be able to 
> run fsck properly (the ones included in EL4 - and EL5 I think - get into an 
> infinite loop at some point while scanning the inode tables).
>
> Currently we try to avoid 'big' ext3 LVs ; the one where we discovered the 
> fsck problems was originally ~6.8TB but we ended up splitting that into 
> several smaller LVs since even with working tools it still took ~2 days to 
> fsck... (and longer to dump/copy/restore it all!)
>
> Some of my co-workers swear by XFS for 'big' volumes but then we do have SGI 
> boxes where XFS (well CXFS) is the expected default fs.  I've not done much 
> testing with XFS on SL mainly because TUV don't like XFS much...

I think it's still the best choice for large (> 2 TB) filesystems. The xfs 
available in SL4 contrib has done very well here. There are some 
interesting effects when such a filesystem runs full and you have to 
remount it with the "inode64" option in order to be able to create new 
files and you discover that quite a few applications are not ready for 
64-bit node numbers. But that aside, it has done very well. No other 
headaches. We're now beginning to deploy large (>10TB) XFS filesystems 
under SL5.

This being said, we now also have lustre OSTs (using a modified ext3) 
7.5 TB in size. No problems so far, but then none of them has run full or 
required an fsck yet.

-- 
Stephan Wiesand
    DESY - DV -
    Platanenallee 6
    15738 Zeuthen, Germany