Jeremy Sanders responded...

|> We can't use that on a file server; we have absolute requirements of more
|> than 32K directory entries at one level in a directory.  This is required
|> by certain applications we use; there's no way around it.
|
|Can you point me to where the limit is stated, or the source code? This 
|page http://answers.google.com/answers/threadview?id=122241 suggests that 
|there is only a limit of 32K subdirectories in a directory and the number 
|of files limit is a lot larger.

But it's the number of directories that is
causing us problems.  We have apps that now
generate 50-60K directories in a directory,
and as chip complexity grows, these apps
will almost certainly create over 100K
directories in a directory.  Not files,
directories.

We observed the problem consistently when
trying to copy such directories from a
NetApp to a Linux:ext3 system, and found
verification all over the internet.  I
ran a test script on SL4 to see if it
still had the problem (we originally
saw this with SL3).  It's still there,
at least in 4.0.  I haven't seen anything
to suggest it's fixed in a newer release.

You can easily test this with a loop set
to create 33,000 directories, and exit
with the number of the directory it's on
when a creation fails.  IIRC the number is
32K less 2.

Documentation?

This is pretty authoritative:

   http://www.kernel-traffic.org/kernel-traffic/kt20010514_118.html
      4. Maximum Number Of Directories In A Directory

Sadly this is from 2001, and nobody seems
to have done anything about it since.  At
least in SL3/RH3 bg_used_dirs_count is
still a __u16, and EXT2_LINK_MAX is still
32000 (in case that matters).

Here's an ext2 internal layout doc:

   http://ftp.fi.muni.cz/pub/gnu/savannah/files/ext2-doc/ext2.pdf

(also from the 2001-2002 time frame).

If you want to look at src, check

   include/linux/ext2_fs.h
   fs/ext2/ialloc.s

2.6.15 source shows the same limit.

[This is one of the reasons I can't quite take
the openfiler project seriously; they haven't
addressed this, either.]

We in the Linux world love to scoff at things
like Bill Gates' and IBM's lack of foresight
in thinking 640KB would be all the mrmory
anyone could want or need, but sometimes we're
just as nearsighted. 8^(