Hi Stijn,

Thank you very much for your reply.

On Sep 9, 2010, at 2:13 AM, Stijn De Weirdt wrote:
> so there is only 1 disk in your system according to dmesg (a 500GB SATA
> disk) and dd reports speeds > 1GB/s. you are either the lucky owner of a
> wonder SATA disk or you are measuring linux caching ;) 

Yes, indeed =).

> when you run the tests, can you also run
> watch grep Dirty /proc/meminfo
> and check if it starts increasing, what the maximum is and when it
> starts decreasing.

It appears to start decreasing at 1GB.  With /proc/sys/vm/dirty_ratio set to 40, the maximum I've seen is 1030708 kB.  With it set to 2, the max I've seen is 991704 kB.

> if you have default SL settings  /proc/sys/vm/dirty_ratio is 40, so
> that's 19GB of dirty memory allowed. with your dd you won't reach that,
> but default /proc/sys/vm/dirty_expire_centisecs is 3000, so after 30
> seconds, the system will start to write the dirty memory away, which is
> probably when the "reduced performance" starts to hit you.
> 
> you can safely lower /proc/sys/vm/dirty_ratio to 2 (which is still
> almost 1GB of dirty memory) to get a more stable performance (i think
> recent kernel do this automated based on total size of memory)
> 
> and if you want to measure your disk performance, use iozone/bonnie/IOR
> with sync options (or add 'sync' to you timing stats)

The system is currently in use and I haven't yet had a chance to run any real disk benchmarks.  For what it's worth, here's the dd output with a subsequent "time sync".  During the sync, the system as a whole becomes sluggish (as we only have a single drive used both for the OS and user data) and it takes several seconds for commands like "top" to return.  We also see comparable behavior when using ext3 or XFS.

[dab66@lnx4103 ~]% time dd if=/dev/zero of=/mnt/scratch/test bs=1M count=1K ; time sync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 0.928643 seconds, 1.2 GB/s

real	0m1.077s
user	0m0.000s
sys	0m1.068s

real	1m35.341s
user	0m0.000s
sys	0m0.182s

> btw, try dstat (sort of iostat and vmstat combined, with colors ;)

Yes, dstat is very nice!

We will let you know whether or not things improve after replacing the disk.

Thanks again,
Devin