Below for the is some performance data taken during initial ZOL evaluation under SL.
**Warning very lengthy read**

Storage HW Setup:
Sun/Oracle Storage 7000 setup. In this case a 7320 (7320 = an x4170m2 w/ dual core westmere and 72GB RAM) w/ one J4410 JBOD hanging off a LSI 9200-8E.
Added were 2 low latency SSD for L2ARC and 2 low latency SSD for ZIL.

HBA is a dual port LSI SAS-2 controller
So 8 Lanes (2 physical SAS2 connections) 2.4 GB/s per connection.
The HBA was slotted into a PCIe Gen 2 (x8 mech/elec) slot so bandwidth = 4 GB/s

Attached JBOD was multipathed through both HBA ports.
In theory we can now saturate the controller and the bus provided we hit the spinning disks at 4GB/s.
The SSD were hung off the dual-port hydra SAS/SATA backplane so again 2.4GB/s x2 bi-directional B/W.

A simple run of FIO: **warning lengthy output below**
-----------------------------------------------------------------------------
----Sequential Reads---- 
Sequential_Reads: (g=0): rw=read, bs=1M-1M/1M-1M, ioengine=sync, iodepth=4 
fio 1.57 
Starting 1 process 
Sequential_Reads: Laying out IO file(s) (1 file(s) / 1000MB) 
Jobs: 1 (f=1): [R] [100.0% done] [4856M/0K /s] [4742 /0  iops] [eta 00m:00s] 
Sequential_Reads: (groupid=0, jobs=1): err= 0: pid=6044 
  read : io=47017MB, bw=4701.3MB/s, iops=4701 , runt= 10001msec 
    clat (usec): min=189 , max=5212 , avg=211.29, stdev=100.66 
     lat (usec): min=189 , max=5212 , avg=211.42, stdev=100.66 
  cpu          : usr=0.79%, sys=97.88%, ctx=1039, majf=0, minf=282 
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 
>=64=0.0% 
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
>=64=0.0% 
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
>=64=0.0% 
     issued r/w/d: total=47017/0/0, short=0/0/0 
     lat (usec): 250=95.32%, 500=4.59%, 750=0.01%, 1000=0.02% 
     lat (msec): 2=0.01%, 4=0.02%, 10=0.04% 
Run status group 0 (all jobs): 
   READ: io=47017MB, aggrb=4701.3MB/s, minb=4814.6MB/s, maxb=4814.6MB/s, 
mint=10001msec, maxt=10001msec 
----Sequential Writes---- 
Sequential_Writes: (g=0): rw=write, bs=1M-1M/1M-1M, ioengine=sync, iodepth=4 
fio 1.57 
Starting 1 process 
Sequential_Writes: Laying out IO file(s) (1 file(s) / 1000MB) 
Jobs: 1 (f=1): [W] [100.0% done] [0K/3397M /s] [0 /3317  iops] [eta 00m:00s] 
Sequential_Writes: (groupid=0, jobs=1): err= 0: pid=5962 
  write: io=30605MB, bw=3060.2MB/s, iops=3060 , runt= 10001msec 
    clat (usec): min=239 , max=4536 , avg=324.80, stdev=136.92 
     lat (usec): min=240 , max=4536 , avg=324.97, stdev=136.96 
    bw (KB/s) : min=1699840, max=1941504, per=58.10%, avg=1820672.00, 
stdev=170882.25 
  cpu          : usr=0.87%, sys=98.53%, ctx=1318, majf=0, minf=23 
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 
>=64=0.0% 
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
>=64=0.0% 
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
>=64=0.0% 
     issued r/w/d: total=0/30605/0, short=0/0/0 
     lat (usec): 250=0.13%, 500=95.66%, 750=2.73%, 1000=0.91% 
     lat (msec): 2=0.41%, 4=0.16%, 10=0.01% 
Run status group 0 (all jobs): 
  WRITE: io=30605MB, aggrb=3060.2MB/s, minb=3133.7MB/s, maxb=3133.7MB/s, 
mint=10001msec, maxt=10001msec 
----Random Reads---- 
Random_Reads: (g=0): rw=randread, bs=1M-1M/1M-1M, ioengine=sync, iodepth=4 
fio 1.57 
Starting 1 process 
Random_Reads: Laying out IO file(s) (1 file(s) / 1000MB) 
Jobs: 1 (f=1): [r] [100.0% done] [4507M/0K /s] [4401 /0  iops] [eta 00m:00s] 
Random_Reads: (groupid=0, jobs=1): err= 0: pid=5987 
  read : io=43589MB, bw=4358.5MB/s, iops=4358 , runt= 10001msec 
    clat (usec): min=198 , max=5103 , avg=227.42, stdev=115.33 
     lat (usec): min=198 , max=5104 , avg=227.54, stdev=115.33 
  cpu          : usr=0.79%, sys=97.93%, ctx=1054, majf=0, minf=280 
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 
>=64=0.0% 
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
>=64=0.0% 
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
>=64=0.0% 
     issued r/w/d: total=43589/0/0, short=0/0/0 
     lat (usec): 250=87.35%, 500=12.50%, 750=0.04%, 1000=0.01% 
     lat (msec): 4=0.06%, 10=0.04% 
Run status group 0 (all jobs): 
   READ: io=43589MB, aggrb=4358.5MB/s, minb=4463.7MB/s, maxb=4463.7MB/s, 
mint=10001msec, maxt=10001msec 
----Random Writes---- 
Random_Writes: (g=0): rw=randwrite, bs=1M-1M/1M-1M, ioengine=sync, iodepth=4 
fio 1.57 
Starting 1 process 
Random_Writes: Laying out IO file(s) (1 file(s) / 1000MB) 
Jobs: 1 (f=1): [w] [100.0% done] [0K/3254M /s] [0 /3178  iops] [eta 00m:00s] 
Random_Writes: (groupid=0, jobs=1): err= 0: pid=5997 
  write: io=30083MB, bw=3007.2MB/s, iops=3007 , runt= 10001msec 
    clat (usec): min=239 , max=5260 , avg=329.82, stdev=158.62 
     lat (usec): min=239 , max=5260 , avg=330.00, stdev=158.65 
    bw (KB/s) : min=1634304, max=1691648, per=53.99%, avg=1662976.00, 
stdev=40548.33 
  cpu          : usr=0.94%, sys=98.40%, ctx=1331, majf=0, minf=22 
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 
>=64=0.0% 
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
>=64=0.0% 
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
>=64=0.0% 
     issued r/w/d: total=0/30083/0, short=0/0/0 
     lat (usec): 250=1.79%, 500=93.27%, 750=3.28%, 1000=0.31% 
     lat (msec): 2=1.20%, 4=0.14%, 10=0.01% 
Run status group 0 (all jobs): 
  WRITE: io=30083MB, aggrb=3007.2MB/s, minb=3080.2MB/s, maxb=3080.2MB/s, 
mint=10001msec, maxt=10001msec 
----Sequential Mixed Reads and Writes---- 
Sequential_Mixed_RW: (g=0): rw=rw, bs=1M-1M/1M-1M, ioengine=sync, iodepth=4 
fio 1.57 
Starting 1 process 
Sequential_Mixed_RW: Laying out IO file(s) (1 file(s) / 1000MB) 
Jobs: 1 (f=1): [M] [27.3% done] [1912M/1846M /s] [1867 /1803  iops] [eta 
00m:08sJobs: 1 (f=1): [M] [36.4% done] [1909M/1807M /s] [1864 /1765  iops] 
[eta 00m:07sJobs: 1 (f=1): [M] [45.5% done] [2024M/1930M /s] [1977 /1885 
iops] [eta 00m:06sJobs: 1 (f=1): [M] [54.5% done] [1898M/1836M /s] [1854 
/1793  iops] [eta 00m:05sJobs: 1 (f=1): [M] [63.6% done] [1485M/1428M /s] 
[1450 /1395  iops] [eta 00m:04sJobs: 1 (f=1): [M] [72.7% done] [1849M/1743M 
/s] [1806 /1702  iops] [eta 00m:03sJobs: 1 (f=1): [M] [81.8% done] 
[2033M/1949M /s] [1985 /1903  iops] [eta 00m:02sJobs: 1 (f=1): [M] [90.9% 
done] [2055M/1968M /s] [2007 /1922  iops] [eta 00m:01sJobs: 1 (f=1): [M] 
[100.0% done] [1933M/1878M /s] [1888 /1834  iops] [eta 00m:00s] 
Sequential_Mixed_RW: (groupid=0, jobs=1): err= 0: pid=6012 
  read : io=17788MB, bw=1778.7MB/s, iops=1778 , runt= 10001msec 
    clat (usec): min=196 , max=4189 , avg=260.84, stdev=154.96 
     lat (usec): min=196 , max=4189 , avg=261.00, stdev=154.97 
    bw (KB/s) : min=763904, max=763904, per=41.94%, avg=763904.00, stdev= 
0.00 
  write: io=17036MB, bw=1703.5MB/s, iops=1703 , runt= 10001msec 
    clat (usec): min=245 , max=15319 , avg=311.01, stdev=199.99 
     lat (usec): min=245 , max=15320 , avg=311.17, stdev=200.01 
    bw (KB/s) : min=782336, max=782336, per=44.85%, avg=782336.00, stdev= 
0.00 
  cpu          : usr=1.12%, sys=97.61%, ctx=1306, majf=0, minf=24 
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 
>=64=0.0% 
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
>=64=0.0% 
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
>=64=0.0% 
     issued r/w/d: total=17788/17036/0, short=0/0/0 
     lat (usec): 250=35.68%, 500=61.27%, 750=1.86%, 1000=0.11% 
     lat (msec): 2=0.95%, 4=0.11%, 10=0.01%, 20=0.01% 
Run status group 0 (all jobs): 
   READ: io=17788MB, aggrb=1778.7MB/s, minb=1821.4MB/s, maxb=1821.4MB/s, 
mint=10001msec, maxt=10001msec 
  WRITE: io=17036MB, aggrb=1703.5MB/s, minb=1744.4MB/s, maxb=1744.4MB/s, 
mint=10001msec, maxt=10001msec 
----Random Mixed Reads and Writes---- 
Random_Mixed_RW: (g=0): rw=randrw, bs=1M-1M/1M-1M, ioengine=sync, iodepth=4 
fio 1.57 
Starting 1 process 
Random_Mixed_RW: Laying out IO file(s) (1 file(s) / 1000MB) 
Jobs: 1 (f=1): [m] [27.3% done] [1820M/1735M /s] [1777 /1694  iops] [eta 
00m:08sJobs: 1 (f=1): [m] [36.4% done] [1920M/1868M /s] [1875 /1824  iops] 
[eta 00m:07sJobs: 1 (f=1): [m] [45.5% done] [2059M/1976M /s] [2011 /1930 
iops] [eta 00m:06sJobs: 1 (f=1): [m] [54.5% done] [1926M/1862M /s] [1881 
/1818  iops] [eta 00m:05sJobs: 1 (f=1): [m] [63.6% done] [1746M/1691M /s] 
[1705 /1651  iops] [eta 00m:04sJobs: 1 (f=1): [m] [72.7% done] [2054M/1979M 
/s] [2006 /1933  iops] [eta 00m:03sJobs: 1 (f=1): [m] [81.8% done] 
[2106M/2031M /s] [2057 /1983  iops] [eta 00m:02sJobs: 1 (f=1): [m] [90.9% 
done] [2096M/2032M /s] [2047 /1984  iops] [eta 00m:01sJobs: 1 (f=1): [m] 
[100.0% done] [1912M/1836M /s] [1867 /1793  iops] [eta 00m:00s] 
Random_Mixed_RW: (groupid=0, jobs=1): err= 0: pid=6037 
  read : io=18250MB, bw=1824.9MB/s, iops=1824 , runt= 10001msec 
    clat (usec): min=200 , max=5959 , avg=251.55, stdev=221.56 
     lat (usec): min=200 , max=5959 , avg=251.70, stdev=221.56 
    bw (KB/s) : min=724992, max=724992, per=38.80%, avg=724992.00, stdev= 
0.00 
  write: io=17600MB, bw=1759.9MB/s, iops=1759 , runt= 10001msec 
    clat (usec): min=245 , max=5941 , avg=302.45, stdev=180.80 
     lat (usec): min=245 , max=5942 , avg=302.62, stdev=180.81 
    bw (KB/s) : min=688128, max=688128, per=38.19%, avg=688128.00, stdev= 
0.00 
  cpu          : usr=0.95%, sys=97.54%, ctx=1278, majf=0, minf=22 
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 
>=64=0.0% 
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
>=64=0.0% 
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
>=64=0.0% 
     issued r/w/d: total=18250/17600/0, short=0/0/0 
     lat (usec): 250=43.72%, 500=54.01%, 750=1.43%, 1000=0.07% 
     lat (msec): 2=0.39%, 4=0.32%, 10=0.06% 
Run status group 0 (all jobs): 
   READ: io=18250MB, aggrb=1824.9MB/s, minb=1868.7MB/s, maxb=1868.7MB/s, 
mint=10001msec, maxt=10001msec 
  WRITE: io=17600MB, aggrb=1759.9MB/s, minb=1802.7MB/s, maxb=1802.7MB/s, 
mint=10001msec, maxt=10001msec
------------------------------------------------------------------------
Running the same under Solaris on the same hardware yielded slower results by ~30 - 40%
------------------------------------------------------------------------

ISCSI/NFS testing
NFS and ISCSI performance was better with ZOL than with the native "Fishworks Solaris" on the above Sun/Oracle storage HW.

Networking setup to Sun Blade 6000 chassis:
Two 10GB ports separate switches, separate subnets attached to seperate NEM ports:
S7K --> 10GB Port --> 10GB Switch --> NEM0 P0 --> HXGE --> x6270 Blade
S7K --> 10GB Port --> 10GB Switch --> NEM0 P1 --> HXGE --> x6270 Blade

On the x6270 I use iscsi + dm_multipath
On the S7K (x4170M2) I use Linux MD Raid + LVM + DM_Multipath + IET (tgtd)

Simple test with FIO yields 1+GB/s throughput via ISCSI with Linux.
With fishworks the performance is much lower ~550-600 MB/s range.

With NFS and Linux ~800 MB/S (here we use bonding)
With fishworks ~350-400  MB/s (here we use solaris aggr)

regards,
Chris Brown
GE Healthcare Technologies
Compute Systems Architect