SCIENTIFIC-LINUX-USERS Archives

July 2013

SCIENTIFIC-LINUX-USERS@LISTSERV.FNAL.GOV
Options: Use Monospaced Font
Show Text Part by Default
Show All Mail Headers

Message: [<< First] [< Prev] [Next >] [Last >>]
Topic: [<< First] [< Prev] [Next >] [Last >>]
Author: [<< First] [< Prev] [Next >] [Last >>]

Print Reply
Subject:
KVM + NFS Performance (ZFS/BTRFS/EXT4)
From:
"Brown, Chris (GE Healthcare)" <[log in to unmask]>
Reply To:
Brown, Chris (GE Healthcare)
Date:
Wed, 3 Jul 2013 13:23:05 -0500
Content-Type:
text/plain
Parts/Attachments:
text/plain (398 lines) 
I have done a fair bit of testing and experimenting to maximize KVM VM
performance on ZFS. Hopefully someone finds the below useful.

Guests/Benchmark used:
WinXP Pro + Crystal Disk Mark 1 pass at 1000MG
SL 6.4 + Gnome Disk Utility (palimpsest) R/W disk benchmark function

Each guest uses a file backed sparse raw disk located on an NFS mounted
directory ("/foo"). The NFS mounted directory is
interconnected via a dual 10Gbe Twin-ax copper from the targeted storage
server disk subsystem and filesystem. The connection is bonded using Linux
bonding (mode=4 miimon=250 xmit_hash_policy=layer2+3). NFS server export
options were (rw,async,no_root_squash). The NFS client was left with default
mount options. The NFS version used was NFSv3 only. NFSv4 was disabled on
the storage  server via "RPCNFSDARGS = -N 4".

*NOTES*
ZFS, EXT4, MD+LVM, or BTRFS disk subsystem configuration details are
specified in each below test configuration.
Low latency Intel SSD models were utilized in all cases wherein SSD are
being used.
When ext4 is used on top of a zvol with default mount options the
journal/metadata is synchronous and therefore offloaded to the ZIL.

WinXP qemu-kvm commandline:
/usr/libexec/qemu-kvm -name winxpfoo -M rhel6.4.0 -cpu qemu64,-svm
-enable-kvm -m 3072 -smp 2,sockets=1,cores=2,threads=1 -nodefconfig
-nodefaults -drive
file=/foo/winxp.img,if=none,id=drive-virtio-disk0,format=raw,cache=none,werror=stop,rerror=stop,aio=threads
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0
-spice port=5900,addr=3.57.109.210,disable-ticketing -k en-us -vga qxl
-global qxl-vga.ram_size=67108864 -global qxl-vga.vram_size=67108864 &

SL 6.4 qemu-kvm commandline:
/usr/libexec/qemu-kvm -name sl64 -M rhel6.4.0 -cpu qemu64,-svm -enable-kvm
-m 3072 -smp 2,sockets=1,cores=2,threads=1 -nodefconfig -nodefaults -drive
file=/vmstore/foo/SL64hd0.img,if=none,id=drive-virtio-disk0,format=raw,cache=none,werror=stop,rerror=stop,aio=threads
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2
-drive
file=/vmstore/foo/SL64hd1.img,if=none,id=drive-virtio-disk1,format=raw,cache=none,werror=stop,rerror=stop,aio=threads
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk1,id=virtio-disk1
-spice port=5900,addr=3.57.109.210,disable-ticketing -k en-us -vga qxl
-global qxl-vga.ram_size=67108864 -global qxl-vga.vram_size=67108864 &

-- Test Config #1 --

Storage Server: Sunfire x4140
RAM: 32GB
CPU: 2x AMD Opteron 2439SE
HBA: LSI 3801
Disks: 8 146GB 10K 2.5" SAS
Disk Config: LSI HW IME Raid  + Linux LVM
Filesystem: LVM+ext4
Network: Intel x520DA2 dual port 10Gbe
OS: SL 6.4

KVM/NFS Client: Sunfire X4170m2
RAM: 72GB RAM
CPU: 2x Intel E5540
HBA: LSI 92118i
Disks: 2 146GB 10K 2.5" SAS
Disk Config: LSI HW IS Raid
Filesystem: LSI HW Raid+ ext4
Network: Intel x520DA2 dual port 10Gbe
OS: OEL 6.4

Benchmark Results (MB/s):
**** WinXP ****
seq: 295.4 189.7
512k: 298.8 188.9
4K: 14.75 12.99
4K QD32: 16.16 14.42

**** SL 6.4  (MB/s) ****
Min Read: 654.3
Max Read: 704.8
Avg Read: 682.5
Min Write: 233.6
Max Write: 551.1
Avg Write: 465.8
AVG Access: 0.5ms

-- End Test Config #1 --

-- Test Config #2 --
Storage Server: Sunfire x4270m2
RAM: 72GB
CPU: 2x Intel E5540
HBA: LSI 92118i
Disks: 24 300GB 10K 2.5" SAS
Disks: 4 128GB Intel SATA SSD
Disk Config: ZFS Raid10 + Raid 0 ZIL + Raid 0 L2ARC
Filesystem: ZFS ZVOL+ext4
Network: Intel x520DA2 dual port 10Gbe
OS: OEL 6.4

KVM/NFS Client: Sunfire X4170m2
RAM: 72GB RAM
CPU: 2x Intel E5540
HBA: LSI 92118i
Disks: 2 146GB 10K 2.5" SAS
Disk Config: LSI HW IS Raid
Network: Intel x520DA2 dual port 10Gbe
OS: OEL 6.4

Filesystem/ZFS Configuration Details/Commands
zpool create das0 mirror <dev> <dev> mirror <dev> <dev>... log <ssd> <ssd>
cache <ssd> <ssd>
zfs create -s -V 100G -o volblocksize=64K das0/foo
mkfs.ext4 -L foo /dev/das0/foo
mount /dev/das0/foo /zvol/foo -o noatime

**** XP (MB/s) ****
seq: 315.4 273.4
512k: 302.4 263.0
4K: 12.94 8.688
4K QD32: 14.45 13.13

**** SL 6.4 (MB/s)****
Min Read: 630.0
Max Read: 698.2
Avg Read: 662.8
Min Write: 357.6
Max Write: 1.1 GB/s
Avg Write: 895.3
AVG Access: 0.5ms

-- End Test Config #2 --

-- Test Config #3 --

Storage Server: Sunfire x4270m2
RAM: 72GB
CPU: 2x Intel E5540
HBA: LSI 92118i
Disks: 24 300GB 10K 2.5" SAS
Disks: 4 128GB Intel SATA SSD
Disk Config: ZFS Raid10 + Raid 0 ZIL + Raid 0 L2ARC
Filesystem: ZFS
Network: Intel x520DA2 dual port 10Gbe
OS: OEL 6.4

KVM/NFS Client: Sunfire X4170m2
RAM: 72GB RAM
CPU: 2x Intel E5540
HBA: LSI 92118i
Disks: 2 146GB 10K 2.5" SAS
Disk Config: LSI HW IS Raid
Network: Intel x520DA2 dual port 10Gbe
OS: OEL 6.4

Filesystem/ZFS Configuration Details/Commands
zpool create das0 mirror <dev> <dev> mirror <dev> <dev>... log <ssd> <ssd>
cache <ssd> <ssd>
zfs create das0/foo

**** XP (MB/s) ****
seq: 111.4 57.36
512k: 95.64 54.33
4K: 11.80 2.878
4K QD32: 11.31 3.095

-- End Test Config #3 --


-- Test Config #4 --

Storage Server: Sunfire x4170
RAM: 72GB
CPU:2x Intel E5540
HBA: LSI 92118i
Disks: 4 200GB Intel SATA SSD
Disks: 2 100 GB Intel SATA SSD
Disk Config: ZFS Raid10 + ZIL + L2ARC
Filesystem: ZFS ZVOL+ext4
Network: Intel x520DA2 dual port 10Gbe
OS: Fedora 18

KVM/NFS Client: Sunfire X4170m2
RAM: 72GB RAM
CPU: 2x Intel E5540
HBA: LSI 92118i
Disks: 2 146GB 10K 2.5" SAS
Disk Config: LSI HW IS Raid
Network: Intel x520DA2 dual port 10Gbe
OS: OEL 6.4

Filesystem/ZFS Configuration Details/Commands
zpool create das0 mirror <dev> <dev> mirror <dev> <dev> log <ssd> cache <ssd>
zfs create -s -V 100G -o volblocksize=64K das0/foo
mkfs.ext4 -L foo /dev/das0/foo
mount /dev/das0/foo /zvol/foo -o noatime

**** XP (MB/s) ****
seq: 196.5 200.2
512k: 195.2 188.8
4K: 9.665 7.043
4K QD32: 12.02 8.140

**** SL 6.4 (MB/s) ****
Min Read: 423.7
Max Read: 575.3
Avg Read: 526.7
Min Write: 44.4
Max Write: 668.8
Avg Write: 540.5
AVG Access: 1.4ms


-- End Test Config #4 --

-- Test Config #5 --

Storage Server: Sunfire x4170
RAM: 72GB
CPU:2x Intel E5540
HBA: LSI 92118i
Disks: 4 200GB Intel SATA SSD
Disks: 2 100 GB Intel SATA SSD
Disk Config: ZFS Raid10 + ZIL + L2ARC
Filesystem: ZFS
Network: Intel x520DA2 dual port 10Gbe
OS: Fedora 18

KVM/NFS Client: Sunfire X4170m2
RAM: 72GB RAM
CPU: 2x Intel E5540
HBA: LSI 92118i
Disks: 2 146GB 10K 2.5" SAS
Disk Config: LSI HW IS Raid
Network: Intel x520DA2 dual port 10Gbe
OS: OEL 6.4

Filesystem/ZFS Configuration Details/Commands
zpool create das0 mirror <dev> <dev> mirror <dev> <dev> log <ssd> cache <ssd>
zfs create das0/foo

**** XP (MB/s) ****
seq: 152.5 126.2
512k: 147.4 121.8
4K: 9.189 7.481
4K QD32: 11.90 8.003

**** SL 6.4 (MB/s) ****
Min Read: 409.3
Max Read: 564.4
Avg Read: 511.6
Min Write: 237.1
Max Write: 658.0
Avg Write: 555.3
AVG Access: 2.9ms

-- End Test Config #5 --


-- Test Config #6 --

Storage Server: Sunfire x4170
RAM: 72GB
CPU:2x Intel E5540
HBA: LSI 92118i
Disks: 4 200GB Intel SATA SSD
Disks: 2 100 GB Intel SATA SSD
Disk Config: Linux MD Raid10 + LVM + ext4
Filesystem: ext4
Network: Intel x520DA2 dual port 10Gbe
OS: OEL 6.4

KVM/NFS Client: Sunfire X4170m2
RAM: 72GB RAM
CPU: 2x Intel E5540
HBA: LSI 92118i
Disks: 2 146GB 10K 2.5" SAS
Disk Config: LSI HW IS Raid
Network: Intel x520DA2 dual port 10Gbe
OS: OEL 6.4

Filesystem/ext4 Configuration Details/Commands
4 Disk SSD  Linux MD Raid 10 Stripe Size 1MB
100GB LV(foo) from VG0 via PV MD device (ext4 data )

2 Disk SSD  Linux MD Raid 0 Stripe Size 1MB
1GB LV(journaldev) from VG1 via PV MD device (ext4 journal/metadata)

ext4 filesystem lv-foo, external journal on lv-journaldev (external
journal_data_ordered)
mounted at /vg0/foo (mount options 
rw,noatime,journal_checksum,journal_async_commit)

**** XP (MB/s) ****
seq: 251.3 203.9
512k: 249.0 217.7
4K: 10.12 9.012
4K QD32: 13.18 11.54

**** SL 6.4 (MB/s) ****
Min Read: 523.3
Max Read: 663.0
Avg Read: 606.4
Min Write: 239.9
Max Write: 656.0
Avg Write: 563.8
AVG Access: 0.8ms

-- End Test Config #6 --

-- Test Config #7 --

Storage Server: Sunfire x4170
RAM: 72GB
CPU:2x Intel E5540
HBA: LSI 92118i
Disks: 4 200GB Intel SATA SSD
Disks: 2 100 GB Intel SATA SSD
Disk Config: BTRFS Raid10 data + raid0 metadata
Filesystem: BTRFS
Network: Intel x520DA2 dual port 10Gbe
OS: Fedora 18

KVM/NFS Client: Sunfire X4170m2
RAM: 72GB RAM
CPU: 2x Intel E5540
HBA: LSI 92118i
Disks: 2 146GB 10K 2.5" SAS
Disk Config: LSI HW IS Raid
Network: Intel x520DA2 dual port 10Gbe
OS: OEL 6.4

Filesystem/BTRFS Configuration Details/Commands
mkfs.btrfs -L foo -m raid0 <ssd> <ssd> -d raid10 <ssd> <ssd> <ssd> <ssd>
btrfs subvolume create /das0/foo

**** XP (MB/s) ****
seq: 195.4 180.0
512k: 180.8 174.5
4K: 9.898 7.183
4K QD32: 11.87 8.248

**** SL 6.4 (MB/s) ****
Min Read: 465.4
Max Read: 554.6
Avg Read: 514.1
Min Write: 450.5
Max Write: 670.3
Avg Write: 525.3
AVG Access: 0.8ms

-- End Test Config #7--


-- Test Config #8 --

Storage Server: Sunfire x4170
RAM: 72GB
CPU:2x Intel E5540
HBA: LSI 92118i
Disks: 4 200GB Intel SATA SSD
Disks: 2 100 GB Intel SATA SSD
Disk Config: BTRFS Raid10 data + raid0 metadata
Filesystem: BTRFS
Network: Intel x520DA2 dual port 10Gbe
OS: OEL 6.4

KVM/NFS Client: Sunfire X4170m2
RAM: 72GB RAM
CPU: 2x Intel E5540
HBA: LSI 92118i
Disks: 2 146GB 10K 2.5" SAS
Disk Config: LSI HW IS Raid
Network: Intel x520DA2 dual port 10Gbe
OS: OEL 6.4

Filesystem/BTRFS Configuration Details/Commands
mkfs.btrfs -L foo -m raid0 <ssd> <ssd> -d raid10 <ssd> <ssd> <ssd> <ssd>
btrfs subvolume create /das0/foo

**** XP (MB/s) ****
seq: 232.1 123.4
512k: 243.0 213.6
4K: 9.393 9.170
4K QD32: 12.04 10.74

**** SL 6.4 (MB/s) ****
Min Read: 508.0
Max Read: 652.4
Avg Read: 594.4
Min Write: 409.6
Max Write: 670.4
Avg Write: 525.8
AVG Access: 0.5

-- End Test Config #8 --

- DHC

ATOM RSS1 RSS2